Re: [Xen-users] Remus: network buffering problem

[Wenda Ni <wonda.ni@xxxxxxxxx>]

Thanx, Shriram.

This is the most confusing part and the biggest worry in my mind. So far, I have not managed to figure out what the difference between PVOPS and HVM  is, particularly to the extent that how I can identify or create a domain with clear view of whether it is PVOPS or HVM. I understand that in CentOS 5.5, the first question of "virt-install --prompt" is on whether your want your domU to be created fully virtualized or not. But when using xen-tools (on Ubuntu), it is not so obvious.

I have an previous post on xen-users discussing my confusion, and soliciting knowledge. Unfortunately, it is still left unanswered so far. The post is attached as follows (with modifications):

I have Xen 4.0.1 pvops on Ubuntu 10.10 (The original Ubuntu kernel is 2.6.35-25-generic) following the steps on https://help.ubuntu.com/community/Xen
I use xen-tools (debootstrap) to configure domU. In xen-tools.config, I specify
"

kernel = /boot/vmlinuz-2.6.18.8
initrd = /boot/initrd.img-2.6.18.8

"
It seems that the actual .cfg file generated for domU uses 

"
bootloader = '/usr/bin/pygrub'

"
There are some explanations about pygrub on xenwiki, so I believe this should be fine. The .cfg created is as follows:

#
# Configuration file for the Xen instance Alice, created
# by xen-tools 4.2 on Mon Feb 21 00:30:22 2011.
#

#
#  Kernel + memory size
#


bootloader = '/usr/bin/pygrub'

vcpus       = '1'
memory      = '128'

#
#  Disk device(s).
#
root        = '/dev/sda2 ro'
disk        = [
                  'file:/mnt/vm-nfs-storage/domains/Alice/disk.img,sda2,w',
                  'file:/mnt/vm-nfs-storage/domains/Alice/swap.img,sda1,w',
              ]


#
#  Physical volumes
#


#
#  Hostname
#
name        = 'Alice'

#
#  Networking
#
vif         = [ 'ip=155.98.38.230,mac=00:16:3E:B1:BE:76' ]

#
#  Behaviour
#
>
on_reboot   = 'restart'
on_crash    = 'restart'

And when the domU starts, the "uname -a" gives kernel as 2.6.35-25-generic, which is exactly the same as the original Ubuntu kernel version.

So my question is 1) how to find whether domU is pv-ops or HVM?


Next, According to http://wiki.xen.org/xenwiki/XenParavirtOps?, I checked the ..../build-linux-2.6-pvops_x86_32/./config,

1. If building 32 bit kernel make sure you have CONFIG_X86_PAE enabled (which is set by selecting CONFIG_HIGHMEM64G)
   • non-PAE mode doesn't work in 2.6.25, and has been dropped altogether from 2.6.26 and newer kernel versions.
2. Enable these core options:
   1. CONFIG_PARAVIRT_GUEST
   2. CONFIG_XEN
3. And Xen pv device support
   1. CONFIG_HVC_DRIVER and CONFIG_HVC_XEN
   2. CONFIG_XEN_BLKDEV_FRONTEND
   3. CONFIG_XEN_NETDEV_FRONTEND

These options are all enabled, so I believe the dom0 built does support pv-ops domU. Then 2) how do I build a pv-ops domU?

I have been struggling to find out the answer, but no progress so far in part due to my limited knowledge on OS. I am new to this area, so please forgive any of my simple questions, and hope the reply can be easy to follow.

On Mon, Feb 21, 2011 at 11:11 AM, Shriram Rajagopalan <rshriram@xxxxxxxxx> wrote:

Yes absolutely. As the Remus wiki in xensource says, it still needs 2.6.18 domU kernel (or novel suse 2.6.27) to operate in fast mode. 

 What type of guest are u running? PV or hvm?

Shriram

On 2011-02-20, at 11:44 PM, Wenda Ni <wonda.ni@xxxxxxxxx> wrote:

Thank you so much, Shriram. This error has been well handled. Now I meet with the following error:

WARNING: suspend event channel unavailable, falling back to slow xenstore signalling
Had 0 unexplained entries in p2m table
 1: sent 32768, skipped 0, delta 1477ms, dom0 38%, target 0%, sent 726Mb/s, dirtied 0Mb/s 0 pages
 2: sent 0, skipped 0, Start last iteration
PROF: suspending at 1298269217.559698
timeout polling fd
ERROR Internal error: Suspend request failed
ERROR Internal error: Domain appears not to have suspended
Save exit rc=1
tc filter del dev vif2.0 parent ffff: proto ip pref 10 u32
RTNETLINK answers: Invalid argument
We have an error talking to the kernel
Exception xen.remus.util.PipeException: PipeException('tc failed: 2, No such file or directory',) in <bound method BufferedNIC.__del__ of <xen.remus.device.BufferedNIC object at 0x228cd90>> ignored

It is related to the guest kernel?

On Sun, Feb 20, 2011 at 11:56 PM, Shriram Rajagopalan <rshriram@xxxxxxxxx> wrote:

On Sun, Feb 20, 2011 at 8:39 PM, Wenda Ni <wonda.ni@gmail.com> wrote:
> Hi Shriram,
>

> I have disabled NET_SCH_PLUG.
>
> I have modified the tools/remus/kmod/Makefile by removing "grep -q
> ^CONFIG_IMQ= $(KERNELDIR)/.config 2>/dev/null" as
>
> XEN_ROOT=../../..
> include $(XEN_ROOT)/tools/Rules.mk
>
> # Should make makefiles export linux build directory!
> # This is a fragile hack to tide us over
> ifeq ($(KERNELS),linux-2.6-xen)
> LINUX_VER=2.6.18-xen
> endif
> ifeq ($(KERNELS),linux-2.6-xen0)
> LINUX_VER=2.6.18-xen0
> endif
> ifeq ($(KERNELS),linux-2.6-pvops)
> LINUX_VER=2.6-pvops
> endif
>
> KERNELDIR ?= $(XEN_ROOT)/build-linux-$(LINUX_VER)_$(XEN_TARGET_ARCH)
>
> .PHONY: all
> all:
>         if test -d $(KERNELDIR); then $(MAKE) -C $(KERNELDIR) SUBDIRS=`pwd`
> modules; fi
>
> .PHONY: install
> install: all
>         if test -d $(KERNELDIR); then $(MAKE) -C $(KERNELDIR) SUBDIRS=`pwd`
> INSTALL_MOD_PATH=$(DESTDIR) modules_install; fi
>
> clean::
>         -rm -rf *.o *.ko *.mod.c *.mod.o Module.symvers .*.cmd .tmp_versions
>

My bad. Just edit the Kbuild file to
obj-m := sch_queue.o

This will prevent the ebt_imq modules from being built, since we are using
IFB (not IMQ). The rest is usual make and make install stuff.

> When I do a "sudo make" in the same directory, it gives the following error.
> Is there a a model called ebt_imq in the menuconfig? Where can i find it? I
> have searched a lot, but failed.
>
> /mnt/local/xen-4.0.1/tools/remus/kmod/ebt_imq.c: In function ‘ebt_imq_tg’:
> /mnt/local/xen-4.0.1/tools/remus/kmod/ebt_imq.c:62: error: ‘struct sk_buff’
> has no member named ‘imq_flags’
> /mnt/local/xen-4.0.1/tools/remus/kmod/ebt_imq.c:62: error: ‘IMQ_F_ENQUEUE’
> undeclared (first use in this function)
> /mnt/local/xen-4.0.1/tools/remus/kmod/ebt_imq.c:62: error: (Each undeclared
> identifier is reported only once
> /mnt/local/xen-4.0.1/tools/remus/kmod/ebt_imq.c:62: error: for each function
> it appears in.)
> make[4]: *** [/mnt/local/xen-4.0.1/tools/remus/kmod/ebt_imq.o] Error 1
> make[3]: *** [_module_/mnt/local/xen-4.0.1/tools/remus/kmod] Error 2
> make[2]: *** [sub-make] Error 2
> make[1]: *** [all] Error 2
> make[1]: Leaving directory
> `/mnt/local/xen-4.0.1/build-linux-2.6-pvops_x86_64'
> make: *** [all] Error 2
>
> You said I need to manually configure the KERNELDIR. Could you be more
> specific? I am new to this area. please make it easy to implement. Thank you
> very much.
>
>
> On Sun, Feb 20, 2011 at 1:10 AM, Shriram Rajagopalan <rshriram@xxxxxxxxx>
> wrote:
>>
>> On Sat, Feb 19, 2011 at 9:14 PM, Wenda Ni <wonda.ni@gmail.com> wrote:
>> > Hi Shriram,
>> >
>> > Thank you so much for your advice.
>> >
>> > I have enabled IFB (Intermediate functional block) module. I have
>> > enabled
>> > Ingress, redirecting, mirroring, NET_SCH_PLUG, and Netfilter. I have
>> > moduled
>> > or enabled the rest of the selections under QoS and/or fair queueing.
>> > Now it
>> > give the following error.
>> >
>> > I cannot find the sch_queue module under QoS and/or fair queueing. I
>> > find a
>> > few previous discussions about the sch_queue module, but have not found
>> > a
>> > good solution. Could you help a bit further? I am using XEN 4.0.1
>> > release
>> > pvops on Ubuntu 10.10 (2.6.32.27).
>> >
>> You only need either sch_plug or the sch_queue. not both.
>>  this issue has been fixed in the latest xen-unstable. But it still
>> lingers in 4.0.1. Here is what I suggest
>>
>> option 1:
>> Go to tools/remus/kmod/
>>  Hack the Makefile. I mean, manually set the KERNELDIR,
>>  remove the "grep CONFIG_IMQ..".
>>  if you do a make now, you should be able to compile the sch_queue module
>>  and then a make install.
>>  Basically, fix the makefile and compile/install the sch_queue module
>>
>> Option 2:
>>  (I suggest you avoid this). Look at device.py and qdisc.py from
>> xen-unstable
>> and fix your local repo's version (from xen 4.0.1).
>>
>> BTW, you are probably going to be raising the next issue - Disk
>> replication fails.
>>  This is a bad hack but probably saves u time, for the moment, in
>> 4.0.1. Edit the device.py near
>>  if disk.uname.startswith("tap:remus..") to "tap2:remus"...
>>  Unfortunately, 4.0.1 decided that users should explicitly specify
>> tap2:... to use blktap2 driver,
>> which meant that using tap:aio leads to loading blktap1
>> driver(deprecated/doesnt work in pvops).
>> So, one would have to use tap2:aio (loads blktap2 driver), and thus
>> similarly tap2:remus:..
>>        - but the remus scripts werent fixed soon enough to reflect
>> this, in 4.0.1. :(
>>
>> shriram
>> >
>> > modprobe -q sch_queue
>> >
>> > Traceback (most recent call last):
>> >   File "/usr/bin/remus", line 210, in <module>
>> >     run(cfg)
>> >   File "/usr/bin/remus", line 119, in run
>> >     bufs.append(BufferedNIC(vif))
>> >   File "/usr/local/lib/python2.6/dist-packages/xen/remus/device.py",
>> > line
>> > 275, in __init__
>> >     self.setup()
>> >   File "/usr/local/lib/python2.6/dist-packages/xen/remus/device.py",
>> > line
>> > 314, in setup
>> >     raise BufferedNICException('could not load sch_queue module')
>> > xen.remus.device.BufferedNICException: could not load sch_queue module
>> >
>> >
>> >
>> > On Sat, Feb 19, 2011 at 6:25 PM, Shriram Rajagopalan
>> > <rshriram@xxxxxxxxx>
>> > wrote:
>> >>
>> >> On 2011-02-18, at 7:04 PM, Wenda Ni <wonda.ni@gmail.com> wrote:
>> >>
>> >> Hi, Todd
>> >>
>> >> I realize what you mean. <330.gif> Hope Shriram can shed some light to
>> >> us!
>> >>
>> >> On Fri, Feb 18, 2011 at 9:44 PM, Todd Deshane <todd.deshane@xxxxxxx>
>> >> wrote:
>> >>>
>> >>> Adding our new remus maintainer to the CC
>> >>>
>> >>> On Fri, Feb 18, 2011 at 6:24 AM, Wenda Ni <wonda.ni@gmail.com> wrote:
>> >>> > Hello,
>> >>> >
>> >>> > Many thanks to Simon for his trouble shooting on live migration. I
>> >>> > have
>> >>> > now
>> >>> > started testing remus according to the remus link
>> >>> > http://nss.cs.ubc.ca/remus/doc.html
>> >>> >
>> >>> > 1) Remus in its simplest mode (--no-net option), without disk
>> >>> > replication or
>> >>> > network protection, is now working good.
>> >>> >
>> >>> > 2) When we add the network buffering protection (remove the --no-net
>> >>> > option), remus gives the error as follows:
>> >>> >
>> >>> > modprobe -q ifb
>> >>> >
>> >>> > modprobe -q imq
>> >>> >
>> >>
>> >> Have you installed the ifb module?
>> >> You have to enable it while configuring the dom0 kernel.
>> >>  Device Drivers-->network devices-->Intermediate Frame Buffer
>> >> You also need to enable a whole bunch of other things
>> >>  Networking ->QoS ->Ingress, actions, mirred, redirect, etc
>> >>   and Plug Queue (NET_SCH_PLUG).
>> >>
>> >> You are better off at the moment to just enable all or most of the
>> >> stuff
>> >> under Netfilter and Qos as modules. Might make your task easier.
>> >>
>> >> Btw thanks for pointing this out. These should probably be enabled by
>> >> the
>> >> kernel build scripts (buildconfigs/enable-xen-config ) or some such
>> >> place.
>> >>
>> >> Shriram
>> >>>
>> >>> > Traceback (most recent call last):
>> >>> >   File "/usr/bin/remus", line 210, in <module>
>> >>> >     run(cfg)
>> >>> >   File "/usr/bin/remus", line 119, in run
>> >>> >     bufs.append(BufferedNIC(vif))
>> >>> >   File "/usr/local/lib/python2.6/dist-packages/xen/remus/device.py",
>> >>> > line
>> >>> > 272, in __init__
>> >>> >     self.pool = Netbufpool(selectnetbuf())
>> >>> >   File "/usr/local/lib/python2.6/dist-packages/xen/remus/device.py",
>> >>> > line
>> >>> > 174, in selectnetbuf
>> >>> >     raise BufferedNICException('no net buffer available')
>> >>> > xen.remus.device.BufferedNICException: no net buffer available
>> >>> > Exception AttributeError: "'BufferedNIC' object has no attribute
>> >>> > 'bufdev'"
>> >>> > in <bound method BufferedNIC.__del__ of
>> >>> > <xen.remus.device.BufferedNIC
>> >>> > object
>> >>> > at 0xd9dd90>> ignored
>> >>> >
>> >>> > I googled, but find little useful info. Could anyone know my
>> >>> > problem?
>> >>> >
>> >>> > --
>> >>> > Cheers,
>> >>> >
>> >>> > Wenda Ni, Ph.D.
>> >>> > Dept. of Computer Science and Engineering
>> >>> > State University of New York (SUNY) at Buffalo
>> >>> >
>> >>> > _______________________________________________
>> >>> > Xen-users mailing list
>> >>> > Xen-users@xxxxxxxxxxxxxxxxxxx
>> >>> > http://lists.xensource.com/xen-users
>> >>> >
>> >>
>> >>
>> >>
>> >> --
>> >> Cheers,
>> >>
>> >> Wenda Ni, Ph.D.
>> >> Dept. of Computer Science and Engineering
>> >> State University of New York (SUNY) at Buffalo
>> >>
>> >
>> >
>> >
>> > --
>> > Cheers,
>> >
>> > Wenda Ni, Ph.D.
>> > Dept. of Computer Science and Engineering
>> > State University of New York (SUNY) at Buffalo
>> >
>> >
>
>
>
> --
> Cheers,
>
> Wenda Ni, Ph.D.
> Dept. of Computer Science and Engineering
> State University of New York (SUNY) at Buffalo
>
>
>
>
> --
> Cheers,
>
> Wenda Ni, Ph.D.
> Dept. of Computer Science and Engineering
> State University of New York (SUNY) at Buffalo
>
>

--

Cheers,

 

Wenda Ni, Ph.D.

Dept. of Computer Science and Engineering

State University of New York (SUNY) at Buffalo

--

Cheers,

 

Wenda Ni, Ph.D.

Dept. of Computer Science and Engineering

State University of New York (SUNY) at Buffalo

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users