RE: [Xen-devel] Xen-unstable panic: FATAL PAGE FAULT

[MaoXiaoyun <tinnycloud@xxxxxxxxxxx>]

Hi Keir:
 
         Thank you for correcting my mistakes.
         Here is the lastest panic and its objdump.
         I am not familiar with assemble language and those regigsters usage.
         I will try to spend some other time to get more understandings.
         What's your opionion?
         btw, the memtest is still running, so far so good, thanks.
 
------------------objdump------------------------------------------------------------------------
 177 ffff82c480115396:<++48 c1 e1 04          <++shl    $0x4,%rcx
 178 ffff82c48011539a:<++4a 03 0c f8          <++add    (%rax,%r15,8),%rcx
 179 }
 180 static inline void
 181 page_list_del(struct page_info *page, struct page_list_head *head)
 182 {
 183     struct page_info *next = pdx_to_page(page->list.next);
 184 ffff82c48011539e:<++8b 03                <++mov    (%rbx),%eax
 185 ffff82c4801153a0:<++48 c1 e0 05          <++shl    $0x5,%rax
 186 ffff82c4801153a4:<++48 29 e8             <++sub    %rbp,%rax< BR> 187 ffff82c4801153a7:<++48 3b 19             <++cmp    (%rcx),%rbx
 188 ffff82c4801153aa:<++0f 84 95 01 00 00    <++je     ffff82c480115545 <free_heap_pages+0x405>
 189     struct page_info *prev = pdx_to_page(page->list.prev);
 190 ffff82c4801153b0:<++89 f2                <++mov    %esi,%edx
 191 ffff82c4801153b2:<++48 c1 e2 05          <++shl    $0x5,%rdx
 192 ffff82c4801153b6:<++48 29 ea             <++sub    %rbp,%rdx
 193 ffff82c4801153b9:<++48 3b 59 08          <++cmp &nbs p;  0x8(%rcx),%rbx
 194 ffff82c4801153bd:<++0f 84 bd 01 00 00    <++je     ffff82c480115580 <free_heap_pages+0x440>
 195
 196     if ( !__page_list_del_head(page, head, next, prev) )
 197     {
 198         next->list.prev = page->list.prev;
 199 ffff82c4801153c3:<++89 70 04             <++mov    %esi,0x4(%rax)
 200         prev->list.next = page->list.next;
 201 ffff82c4801153c6:<++8b 03                <++mov    (%rbx),%eax                      &nbs p;                                                                  
 202 ffff82c4801153c8:<++89 02                <++mov    %eax,(%rdx)
 203 ffff82c4801153ca:<++49 89 dd             <++mov    %rbx,%r13
 204 ffff82c4801153cd:<++41 83 c4 01          <++add    $0x1,%r12d
 205 ffff82c4801153d1:<++41 83 fc 12          <++cmp   ;  $0x12,%r12d
 206 ffff82c4801153d5:<++0f 84 e3 00 00 00    <++je     ffff82c4801154be <free_heap_pages+0x37e>
 207 ffff82c4801153db:<++48 bd 00 00 00 00 0a <++mov    $0x7d0a00000000,%rbp
 208 ffff82c4801153e2:<++7d 00 00
 209 ffff82c4801153e5:<++44 89 e1             <++mov    %r12d,%ecx
 210 ffff82c4801153e8:<++be 01 00 00 00       <++mov    $0x1,%esi
 
 
---------------------------------------------------------------------------------------------------
blktap_sysfs_create: adding attributes for dev ffff880239496c00
(XEN) ----[ Xen-4.0.0  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    2
(XEN) RIP:    e008:[<ffff82c4801153c3>] free_heap_pages+0x283/0x4a0
(XEN) RFLAGS: 0000000000010282   CONTEXT: hypervisor
(XEN) rax: ffff8315ffffffe0   rbx: ffff82f6093b0040   rcx: ffff83063fc01a20
(XEN) rdx: ffff8315ffffffe0   rsi: 00000000ffffffff   rdi: 000000000049d802
(XEN) rbp: 00007d0a00000000   rsp: ffff83023ff37cb8   r8:  0000000000000000
(XEN) r9:  ffffffffffffffff   r10: ffff83060a3c0018   r11: 0000000000000282
(XEN) r12: 0000000000000000   r13: ffff82f6093b0060   r14: 00000000000001a2
(XEN) r15: 0000000000000001   cr0: 000000008005003b   cr4: 00000000000026f0
(XEN) cr3: 000000008da54000   cr2: ffff83 15ffffffe4
(XEN) ds: 0000   es: 0000   fs: 0063   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff83023ff37cb8:
(XEN)    ffff82f6093b7f80 00000000ffffffe0 00000000000001a2 ffff83060a3c0000
(XEN)    0000000000000000 0000000000000001 ffff82f6093b0060 0000000000000000
(XEN)    ffff82f6093b0080 ffff82c480115732 00000001093b7cc0 ffff82f6093b0060
(XEN)    ffff83060a3c0018 0000000000000000 ffff83060a3c0000 ffff83060a3c0fa8
(XEN)    0000000000000000 ffff82c48014aaa6 ffff83060a3c0fa8 ffff83060a3c0fa8
(XEN)    ffff83060a3c0014 4000000000000000 ffff83023ff37f28 ffff83060a3c0018
(XEN)    0000000000000000 ffff83060a3c0000 0000000000305000 0000000000000009
(XEN)    0000000000000009 ffff82c48014b2fd 00ffffffffffffff ffff83060a3c0000
(XEN)    0000000000000000 ffff83023ff37e28 0000000000305000 ffff82c480105fe0
(XEN)    ffff82c480255240 fffffffffffffff3 0000000002599000 ffff82c4801043ce
(XEN)    ffff82c4801447da 0000000000000080 ffff83023ff37f28 0000000000000096
(XEN)    ffff83023ff37f28 00000000000000fc 0000000600000002 00000000023c0031
(XEN)    0000000000000001 00000039890a8e2a 0000003000000018 000000004523af30
(XEN)    000000004523ae70 0000000000000000 00007fc608ea8a70 000000398903c8a4
(XEN)    000000004523af44 0000000000000000 000000004523b158 0000000000000000
(XEN)    0000007f024f6d20 00007fc60a094750 000000000255ff40 00007fc607be5ea8
(XEN)    fffffffffffffff5 0000000000000246 00000039880cc557 0000000000000100
(XEN)    00000039880cc557 0000000000000033 0000000000000246 ffff8300bf562000
(XEN)    ffff8801db8d3e78 000000004523aec0 0000000000305000 000000 0000000009
(XEN)    0000000000000009 ffff82c4801e3169 0000000000000009 0000000000000009
(XEN) Xen call trace:
(XEN)    [<ffff82c4801153c3>] free_heap_pages+0x283/0x4a0
(XEN)    [<ffff82c480115732>] free_domheap_pages+0x152/0x380
(XEN)    [<ffff82c48014aaa6>] relinquish_memory+0x186/0x530
(XEN)    [<ffff82c48014b2fd>] domain_relinquish_resources+0x1ad/0x280
(XEN)    [<ffff82c480105fe0>] domain_kill+0x80/0xf0
(XEN)    [<ffff82c4801043ce>] do_domctl+0x1be/0x1000
(XEN)    [<ffff82c4801447da>] __find_next_bit+0x6a/0x70
(XEN)    [<ffff82c4801e3169>] syscall_enter+0xa9/0xae
(XEN)   
(XEN) Pagetable walk from ffff8315ffffffe4:
(XEN)  L4[0x106] = 00000000bf569027 5555555555555555
(XEN)  L3[0x057] = 0000000000000000 ffffffffffffffff
(XE N)
(XEN) ****************************************
(XEN) Panic on CPU 2:
(XEN) FATAL PAGE FAULT
(XEN) [error_code=0002]
(XEN) Faulting linear address: ffff8315ffffffe4
(XEN) ****************************************
(XEN)
(XEN) Manual reset required ('noreboot' specified)
 
---------------------------------------------------------------------------------------------------
> Date: Mon, 30 Aug 2010 14:16:09 +0100
> Subject: Re: [Xen-devel] Xen-unstable panic: FATAL PAGE FAULT
> From: keir.fraser@xxxxxxxxxxxxx
> To: tinnycloud@xxxxxxxxxxx; xen-devel@xxxxxxxxxxxxxxxxxxx
>
> On 30/08/2010 14:03, "MaoXiaoyun" <tinnycloud@xxxxxxxxxxx> wrote:
>
> > Appreciate for the quick response.
> >
> > Actually I have done some decode on the backtrace last Friday.
> > According the RIP ffff82c4801153c3, I cut the "objdump -dS xen-syms"
> > (please see below). It looks like the bug happened on the domain page list
>
> ffff82c4801153c3 isn't the start of an instruction in your below
> disassembly. Hence you didn't disassemble exactly the build of Xen which
> crashed. It needs to be exactly the same image.
>
> -- keir
>
> & gt; travels, which is beyond my understanding. Since in my understanding,
> > those domain pages come from kernel memory zone, they are always
> > reside in the physical memory, and the address shouldn't have the chance
> > to be changed, right?
> > If so, what is the relationship between all those panic and free_heap_pages?
> >
> > Several servers (at least 3) experienced the same panic on the same test.
> > Those servers have the identical hardware, kernel and xen configuration.
> > Right now, on one server, memtest is running, shall be finished in a few
> > hours.
> > (24G memory)
> >
> > ------------------------------------------------------------------------------
> > ------
> > 169 static inline void
> > 170 page_list_del(struct page_info *page, struct page_list_head *head)
> > 171 {
> > 172 struct page_info *next = p dx_to_page(page->list.next);
> > 173 struct page_info *prev = pdx_to_page(page->list.prev);
> > 174 ffff82c4801153b8:<++8b 73 04 <++mov 0x4(%rbx),%esi
> > 175 ffff82c4801153bb:<++49 8d 0c 06 <++lea (%r14,%rax,1),%rcx
> > 176 ffff82c4801153bf:<++48 8d 05 fa 10 26 00 <++lea 2494714(%rip),%rax
> > # ffff82c4803764c0 <_heap>
> > 177 ffff82c4801153c6:<++48 c1 e1 04 <++shl $0x4,%rcx
> > 178 ffff82c4801153ca:<++4a 03 0c f8 <++add (%rax,%r15,8),%rcx
> > 179 }
> > 180 static inline void
> > 181 page_list_del(struct page_info *page, struct page_list_head *head)
> > 182 {
> > 183 struct page_info *next = pdx_to_page(page->list.next);
> > 184 ffff82c4801153ce:<++8b 03 <++mov (%rbx),%eax
> > 185 ffff82c4801153d0:<++48 c1 e0 05 <++shl $0x5,%rax
> > 186 ffff82c4801153d4:<++48 29 e8 <++sub %rbp,%r ax
> > 187 ffff82c4801153d7:<++48 3b 19 <++cmp (%rcx),%rbx
> > 188 ffff82c4801153da:<++0f 84 95 01 00 00 <++je ffff82c480115575
> > <free_heap_pages+0x405>
> > 189 struct page_info *prev = pdx_to_page(page->list.prev);
> > 190 ffff82c4801153e0:<++89 f2 <++mov %esi,%edx
> > 191 ffff82c4801153e2:<++48 c1 e2 05 <++shl $0x5,%rdx
> > 192 ffff82c4801153e6:<++48 29 ea <++sub %rbp,%rdx
> > 193 ffff82c4801153e9:<++48 3b 59 08 <++cmp 0x8(%rcx),%rbx
> > 194 ffff82c4801153ed:<++0f 84 bd 01 00 00 <++je ffff82c4801155b0
> > <free_heap_pages+0x440>
> > 195
> > 196 if ( !__page_list_del_head(page, head, next, prev) )
> > 197 {
> > 198
> > ------------------------------------------------------------------------------
> > ------
> >
> >> Date: Mon, 30 Aug 2010 10:02:05 +01 00
> >> Subject: Re: [Xen-devel] Xen-unstable panic: FATAL PAGE FAULT
> >> From: keir.fraser@xxxxxxxxxxxxx
> >> To: tinnycloud@xxxxxxxxxxx; xen-devel@xxxxxxxxxxxxxxxxxxx
> >>
> >> On 30/08/2010 09:47, "MaoXiaoyun" <tinnycloud@xxxxxxxxxxx> wrote:
> >>
> >>> 3) Every panic pointer to the same address: ffff8315ffffffe4, which is
> >>> not a valid page address.
> >>> I printted pages of the domain in assign_pages, which all looks like
> >>> ffff82f60bd64000, at least
> >>> ffff82f60 is the same.
> >>
> >> Yes, well you may not be crashing on a supposed page address. Certainly the
> >> page pointer that relinquish_memory() is working on, and passed to
> >> put_page->free_domheap_pages is valid enough to not cause any of those
> >> functions to crash when dereferenci ng it. At the moment you really have no
> >> idea what is causing free_heap_pages() to crash.
> >>
> >>> A bit of lost direction to go further. Thanks.
> >>
> >> You need to find out which line of code in free_heap_pages() is crashing,
> >> and what variable it is trying to dereference when it crashes. You have a
> >> nice backtrace with an EIP value, so you can 'objdump -d xen-syms' and
> >> search for the EIP in the disassembly. If you have a debug build of Xen you
> >> can even do 'objdump -S xen-syms' and have the disassembly annotated with
> >> corresponding source lines.
> >>
> >> Have you seen this on more than one physical machine? If not, have you run
> >> memtest on the offending machine?
> >>
> >> -- Keir
> >>
> >>
> >
>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel