[Xen-users] Xen, Memory and prelink interaction

[Massimo Mongardini <massimo.mongardini@xxxxxxxxx>]

Hi all,
I am experiencing the below error systematically on all the servers 
where I have installed Xen and reverted back to a redhat kernel.
I managed to fix it but unfortunately I can't understand the reason (not 
yet at least ;) ).
I suspect some strange interaction between Xen kernel and prelink that 
is carried on even after the xen kernel is gone.
The servers are HP ProLiant BL465c G1
cpus: 2xDual-Core AMD Opteron(tm) Processor 2216 HE
os: RHEL 4.6 32bit version (Linux server-0007 2.6.9-67.0.15.ELsmp #1 SMP 
Tue Apr 22 13:50:33 EDT 2008 i686 athlon i386 GNU/Linux)
ram: 8GB

I have several java apps that start with -Xmx2048m option and they work 
just fine on 12 blades, but on some of them that were used as dom0s.
[root@server-0007 jdk1.5.0_11]# ./bin/java -Xmx2048M -version
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

the limit seems to be at 1538m:
sh-3.00# ./bin/java -Xmx1538m -version
java version "1.5.0_11"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_11-b03)
Java HotSpot(TM) Server VM (build 1.5.0_11-b03, mixed mode)

sh-3.00# ./bin/java -Xmx1539m -version
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

These are the steps that make my servers act this way:
- Enabled Virtualization support on the CPU
- Install Xen and a Xen kernel ( I had 3.1.0-1 installed with the rhel4 
rpms from xen.org, not sure if the version matters )
- moved /lib/tls /lib/tls.disabled
- Booted an HVM domU
- Uninstalled Xen Xen Kernel
- Disabled Virtualization support on CPU
- moved /lib/tls.disabled /lib/tls
- Rebooted to a redhat kernel (2.6.9-67.0.15.ELsmp on my case)

Now, there's seems to be plenty of free memory but java fails as above
[massimo@server-0007 ~]$ free -m
            total       used       free     shared    buffers     cached
Mem:          8114        169       7944          0         15         74
-/+ buffers/cache:         79       8035
Swap:            0          0          0

ulimit is not guilty in this:
[massimo@server-0007 ~]$ ulimit -m -v
max memory size         (kbytes, -m) unlimited
virtual memory          (kbytes, -v) unlimited

selinux is set in permissive mode

To fix this problem I did:
(since I spent several hours and many commands on this I am not sure if 
some steps can be skipped. Maybe on the next server that I will switch 
from xen to rh kernel I'll try to shorten the list and maybe give it 
more sense!)
- reboot in "emergency" mode and with selinux=0, (not single user. It 
didn't work in single user mode) ( emergency = nothing is mounted except 
/ in ro)
- mount -t proc none /proc
- mount -o remount,rw /
- prelink -au (undo any prelinking)
- rm -f /etc/prelink.cache
- ldconfig -v
- prelink -avmR (redo prelinking)
- ldconfig -v

[root@(none) jdk1.5.0_11]# ./bin/java -Xmx2048m -version
java version "1.5.0_11"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_11-b03)
Java HotSpot(TM) Server VM (build 1.5.0_11-b03, mixed mode)
## ypeee!

I am sorry if some commands are probably redundant here, I just copied 
and pasted my notes...
So my questions are:
Why is this happening after Xen touched my system?
and
Why in emergency mode?
and ....

Can anyone share some light on this or perhaps try and reproduce it? 
Maybe there are not so many people switching from Xen kernel to Redhat 
kernels that need to run this sort of java applications but I have the 
feeling that there's something buggish behind this...
Regards,
Massimo

--
Massimo Mongardini
~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~
echo 
'Jg!J!hjwf!zpv!bo!bqqmf!boe!zpv!hjwf!nf!bo!bqqmf-!uifo!xf!xjmm!ibwf!bo!bqqmf!fbdi/!Cvu!jg!J!hjwf!zpv!bo!jefb!boe!zpv!hjwf!nf!bo!jefb-!xf!xjmm!ibwf!uxp!jefbt!fbdi!'
 | perl -pe 's/(.)/chr(ord($1)-1)/ge'
~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~
http://massimo.mongardini.it
http://www.getthefacts.it
http://www.mongardini.it/pizza-howto
~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~
Please avoid sending me Word or PowerPoint attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users