Re: [Xen-users] 1000 Domains: Not able to access Domu via xm console from Dom0

[Ian Campbell <Ian.Campbell@xxxxxxxxxx>]

On Thu, 2012-12-13 at 12:24 +0000, Paul Harvey wrote:
> So, i attached strace to xenconsoled to see i could find what was
> going on and i got this
> 
> ioctl(1023, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon
> echo ...}) = 0
> ioctl(1023, TIOCGPTN, [345])            = 0
> stat("/dev/pts/345", {st_mode=S_IFCHR|0620, st_rdev=makedev(136,
> 345), ...}) = 0
> open("/dev/pts/345", O_RDWR|O_NOCTTY)   = -1 EMFILE (Too many open
> files)
> close(1023)                             = 0
> write(2, "Failed to create tty for domain-"..., 70) = 70
> open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 1023
> fstat(1023, {st_mode=S_IFREG|0644, st_size=3661, ...}) = 0
> fstat(1023, {st_mode=S_IFREG|0644, st_size=3661, ...}) = 0
> 
> 
> So this is definitely a problem with file limits, but i don't
> understand as the current limit on files per process is 65000

I wrote the following yesterday and although I see it in my sent box I
can't see it in the list archives and you don't seem to have received it
either. I've no idea where it got to...


On Tue, 2012-12-11 at 22:07 +0000, Paul Harvey wrote:
> On 7 December 2012 10:03, Ian
> Campbell <Ian.Campbell@xxxxxxxxxx> wrote:
>         On Thu, 2012-12-06 at 23:27 +0000, Paul Harvey wrote:
>         
>         > Any help, or is this a limitation of Xen?
>         
>         
>         One limit you might be hitting is the number of event channels
>         which
>         dom0 can handle. The maximum is currently 1024 for a 32-bit
>         domains and
>         4096 for 64-bit (that's per domains, not total in the system).
>         Depending
>         on the configuration of the mini-os domains (e.g. number of
>         devices etc)
>         you might be hitting this -- "lsevtchn 0" might give a clue if
>         this is
>         happening (that tool is in /usr/lib/xen somewhere).
>         
>         Work has just started on expanding these limits to ~32k and
>         ~512k for
>         32- and 64-bit domains respectively, the hope is that this
>         will be done
>         in time for 4.3. Look for posts from Wei Liu on xen-devel this
>         week.
>         
>         If you aren't hitting the evtchn limits then maybe you are
>         hitting some
>         dom0 OS level limitation, i.e. a ulimit on the number of open
>         file
>         descriptors which xenconsoled can have or some limit on the
>         number of
>         pty's.
>         
>         Ian.
> 
> 
> Hi Ian, 
> 
> 
> Thanks for the quick reply!
> 
> 
> Have looked into your suggestions and:
> 
> 
> * It is NOT the number of evntchns, this is much less that the limits
> you mention

OOI how many event channels do your 1000 domains require?

> * It is NOT the number of allowable PTY's, the number used is much
> less than the limit

Again OOI how many?

> * The number of per process file descriptors was set to 1024, but i
> have increased this to thousands : 
> ulimint -n
> 10240

Did you apply this to the xenconsoled and other daemon processes too?
setting ulimit only effects the current process and its children.

> To hammer this point home, i built a wee C file to allocate pty's.
> Before i changed the limit i got problems past 1024, now it work fine
> as root, or any user.
> 
> 
> But, when i create ~350 domains: 
> 
> 
> cat /proc/<xenconsoled>/fd | wc -l  
> 1024
> 
> 
> only ever goes as high as 1024, and does not increase for subsequently
> added domains.

I suspect you haven't actually increased the ulimit for this process.
What does /proc/<xenconsoled>/limits contain?

There may also be sysctls which limit the number of fds a process can
have.

> Any other ideas?

> Also, as a side note, any idea why the domain creation time grows
> quadratically?

Grows with the number of running domains you mean?

There were some memory allocator optimisations discussed on xen-devel
recently, but I don't recall the details enough to know if it is
relevant here, it could be that though. Other than that I'm afraid I've
no ideas.

Ian.


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users