After applying the patch we were able to reproduce the panic and the additional
debugging output is attached. The driver appears to re-try the request several
times before dying:
Writing inode tables: ------------[ cut here ]------------
WARNING: at drivers/block/xen-blkfront.c:244 do_blkif_request+0x301/0x440()
Modules linked in:
Pid: 0, comm: swapper Not tainted 2.6.28.2-metacarta-appliance-1 #2
Call Trace:
<IRQ> [<ffffffff80240b34>] warn_on_slowpath+0x64/0xa0
[<ffffffff80232ae3>] enqueue_task+0x13/0x30
[<ffffffff8059be54>] _spin_unlock_irqrestore+0x14/0x20
[<ffffffff803c70fc>] get_free_entries+0xbc/0x2a0
[<ffffffff804078b1>] do_blkif_request+0x301/0x440
[<ffffffff8036fb35>] blk_invoke_request_fn+0xa5/0x110
[<ffffffff80407a08>] kick_pending_request_queues+0x18/0x30
[<ffffffff80407bb7>] blkif_interrupt+0x197/0x1e0
[<ffffffff8026ccd9>] handle_IRQ_event+0x39/0x80
[<ffffffff8026f096>] handle_level_irq+0x96/0x120
[<ffffffff802140d5>] do_IRQ+0x85/0x110
[<ffffffff803c83f5>] xen_evtchn_do_upcall+0xe5/0x130
[<ffffffff80246217>] __do_softirq+0xe7/0x180
[<ffffffff8059c65e>] xen_do_hypervisor_callback+0x1e/0x30
<EOI> [<ffffffff802093aa>] _stext+0x3aa/0x1000
[<ffffffff802093aa>] _stext+0x3aa/0x1000
[<ffffffff8020de8c>] xen_safe_halt+0xc/0x20
[<ffffffff8020c1fa>] xen_idle+0x2a/0x50
[<ffffffff80210041>] cpu_idle+0x41/0x70
---[ end trace 107c74ebf2b50a63 ]---
METACARTA: too many segments for ring (11): req->nr_phys_segments = 11
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 1536 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 2048 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 2560 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 3072 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 3584 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 0 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 512 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 1024 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 1536 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 2048 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 2560 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 3072 len 512
------------[ cut here ]------------
WARNING: at drivers/block/xen-blkfront.c:244 do_blkif_request+0x301/0x440()
Modules linked in:
Pid: 0, comm: swapper Tainted: G W 2.6.28.2-metacarta-appliance-1 #2
Call Trace:
<IRQ> [<ffffffff80240b34>] warn_on_slowpath+0x64/0xa0
[<ffffffff8059be54>] _spin_unlock_irqrestore+0x14/0x20
[<ffffffff803c70fc>] get_free_entries+0xbc/0x2a0
[<ffffffff804078b1>] do_blkif_request+0x301/0x440
[<ffffffff80407bb7>] blkif_interrupt+0x197/0x1e0
[<ffffffff8026ccd9>] handle_IRQ_event+0x39/0x80
[<ffffffff8026f096>] handle_level_irq+0x96/0x120
[<ffffffff802140d5>] do_IRQ+0x85/0x110
[<ffffffff803c83f5>] xen_evtchn_do_upcall+0xe5/0x130
[<ffffffff80246217>] __do_softirq+0xe7/0x180
[<ffffffff8059c65e>] xen_do_hypervisor_callback+0x1e/0x30
<EOI> [<ffffffff802093aa>] _stext+0x3aa/0x1000
[<ffffffff802093aa>] _stext+0x3aa/0x1000
[<ffffffff8020de8c>] xen_safe_halt+0xc/0x20
[<ffffffff8020c1fa>] xen_idle+0x2a/0x50
[<ffffffff80210041>] cpu_idle+0x41/0x70
---[ end trace 107c74ebf2b50a63 ]---
METACARTA: too many segments for ring (11): req->nr_phys_segments = 11
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 1536 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 2048 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 2560 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 3072 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 3584 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 0 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 512 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 1024 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 1536 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 2048 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 2560 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 3072 len 512
------------[ cut here ]------------
WARNING: at drivers/block/xen-blkfront.c:244 do_blkif_request+0x301/0x440()
Modules linked in:
Pid: 0, comm: swapper Tainted: G W 2.6.28.2-metacarta-appliance-1 #2
Call Trace:
<IRQ> [<ffffffff80240b34>] warn_on_slowpath+0x64/0xa0
[<ffffffff80232ae3>] enqueue_task+0x13/0x30
[<ffffffff8059be54>] _spin_unlock_irqrestore+0x14/0x20
[<ffffffff803c70fc>] get_free_entries+0xbc/0x2a0
[<ffffffff804078b1>] do_blkif_request+0x301/0x440
[<ffffffff8036fb35>] blk_invoke_request_fn+0xa5/0x110
[<ffffffff80407a08>] kick_pending_request_queues+0x18/0x30
[<ffffffff80407bb7>] blkif_interrupt+0x197/0x1e0
[<ffffffff8026ccd9>] handle_IRQ_event+0x39/0x80
[<ffffffff8026f096>] handle_level_irq+0x96/0x120
[<ffffffff802140d5>] do_IRQ+0x85/0x110
[<ffffffff803c83f5>] xen_evtchn_do_upcall+0xe5/0x130
[<ffffffff80246217>] __do_softirq+0xe7/0x180
[<ffffffff8059c65e>] xen_do_hypervisor_callback+0x1e/0x30
<EOI> [<ffffffff802093aa>] _stext+0x3aa/0x1000
[<ffffffff802093aa>] _stext+0x3aa/0x1000
[<ffffffff8020de8c>] xen_safe_halt+0xc/0x20
[<ffffffff8020c1fa>] xen_idle+0x2a/0x50
[<ffffffff80210041>] cpu_idle+0x41/0x70
---[ end trace 107c74ebf2b50a63 ]---
METACARTA: too many segments for ring (11): req->nr_phys_segments = 11
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 1536 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 2048 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 2560 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 3072 len 512
METACARTA: 0: bio page ffffe2000c291d00 pfn 379760 off 3584 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 0 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 512 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 1024 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 1536 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 2048 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 2560 len 512
METACARTA: 0: bio page ffffe2000c291d38 pfn 379761 off 3072 len 512
------------[ cut here ]------------
WARNING: at drivers/block/xen-blkfront.c:244 do_blkif_request+0x301/0x440()
Modules linked in:
Pid: 0, comm: swapper Tainted: G W 2.6.28.2-metacarta-appliance-1 #2
Call Trace:
<IRQ> [<ffffffff80240b34>] warn_on_slowpath+0x64/0xa0
[<ffffffff8059be54>] _spin_unlock_irqrestore+0x14/0x20
[<ffffffff803c70fc>] get_free_entries+0xbc/0x2a0
[<ffffffff804078b1>] do_blkif_request+0x301/0x440
[<ffffffff80407bb7>] blkif_interrupt+0x197/0x1e0
[<ffffffff8026ccd9>] handle_IRQ_event+0x39/0x80
[<ffffffff8026f096>] handle_level_irq+0x96/0x120
[<ffffffff802140d5>] do_IRQ+0x85/0x110
[<ffffffff803c83f5>] xen_evtchn_do_upcall+0xe5/0x13
We also attempted changing the blk_queue_max_*_segments calls to use
BLKIF_MAX_SEGMENTS_PER_REQUEST - 1 and our spinner was able to run overnight
without any panics...
---
Greg Harris
System Administrator
MetaCarta, Inc.
(O) +1 (617) 301-5530
(M) +1 (781) 258-4474
----- "Jeremy Fitzhardinge" <jeremy@xxxxxxxx> wrote:
> Jens Axboe wrote:
> > To shed some more light on this, I'd suggest changing that BUG_ON()
> to
> > some code that simply dumps each segment (each bvec in the iterator
> > list) from start to finish along with values of
> > request->nr_phys_segments and size info.
> >
>
> OK, something like this?
>
> J
>
> Subject: xen/blkfront: try to track down over-segment BUG_ON in
> blkfront
>
> Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@xxxxxxxxxx>
> ---
> drivers/block/xen-blkfront.c | 24 +++++++++++++++++++++++-
> 1 file changed, 23 insertions(+), 1 deletion(-)
>
> ===================================================================
> --- a/drivers/block/xen-blkfront.c
> +++ b/drivers/block/xen-blkfront.c
> @@ -240,7 +240,10 @@
>
> ring_req->nr_segments = 0;
> rq_for_each_segment(bvec, req, iter) {
> - BUG_ON(ring_req->nr_segments == BLKIF_MAX_SEGMENTS_PER_REQUEST);
> + if (WARN_ON(ring_req->nr_segments >=
> + BLKIF_MAX_SEGMENTS_PER_REQUEST))
> + goto dump_req;
> +
> buffer_mfn = pfn_to_mfn(page_to_pfn(bvec->bv_page));
> fsect = bvec->bv_offset >> 9;
> lsect = fsect + (bvec->bv_len >> 9) - 1;
> @@ -274,6 +277,25 @@
> gnttab_free_grant_references(gref_head);
>
> return 0;
> +
> +dump_req:
> + {
> + int i;
> +
> + printk(KERN_DEBUG "too many segments for ring (%d): "
> + "req->nr_phys_segments = %d\n",
> + BLKIF_MAX_SEGMENTS_PER_REQUEST, req->nr_phys_segments);
> +
> + i = 0;
> + rq_for_each_segment(bvec, req, iter) {
> + printk(KERN_DEBUG
> + " %d: bio page %p pfn %lx off %u len %u\n",
> + i++, bvec->bv_page, page_to_pfn(bvec->bv_page),
> + bvec->bv_offset, bvec->bv_len);
> + }
> + }
> +
> + return 1;
> }
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
|
Home