Multi-pointer Remote Desktop
Posted by Chris Ball Mon, 26 Jan 2009 05:10:00 GMT
Remote desktop is a pretty useful technology. It would be more useful if it wasn't so competitive, though; when someone joins, the sharer has to sit back and watch, or fight over who gets control. Well, now it's cooperative — here's a video demo:
Download the video (23M)
I should make it clear that I'm responsible only for the smallest piece of this work. Peter Hutterer's MPX is the core technology enabling an X desktop to have more than one pointer and keyboard present on it, and my work was modifying the Vino VNC server to make it understand how MPX works. As shown in the video, this requires no modifications to VNC clients — each client can become another pointer on your X desktop regardless of whether it's running on a Windows, Mac or other Unix machine, which I think makes the whole thing pretty compelling.
My patches against Vino are here, and this is a complete summary of them: for each new client that joins, we create a new keyboard/pointer pair and store its details with that client, then when new events come in for a client we send them through to the appropriate device. That's all. This is still prototype code, so there are bugs, the most noticeable of which is when you have more than one client connected, each client can only see the pointer of themselves and the server, although the server can see everyone's pointer; this will be fixed by having vino_cursor_update_timeout() and parents act on a list of cursor positions rather than a single one. (Ideally, GDK should become aware of multiple pointers, too.)
The next steps are to fix some bugs and get the code upstream into the main Vino repository, so that it's ready for when the distros start shipping a multi-pointer X server — one way to do this would be to have Vino's preferences check to see whether your X server supports MPX, and ask you to choose between sharing your main pointer or granting extra pointers to new clients if it does.
The original motivation for this work, besides it being a useful-sounding idea in general, was as an experiment for a collaboration model for OLPC. A very intuitive way for writing collaborative applications would be to make programs that don't have to understand anything about networking, but do have the idea that there's going to be more than one pointer and keyboard around sometimes and that they should be treated differently, just like when using a games console with multiple controllers.
For example: to write a collaborative text editor using this model, you'd want to color the text written by each participant's keyboard differently, but other than that you don't need to care about any details of networking. (You'd want an out-of-band way for each participant to get a copy of the file they were working on afterwards, though...)
I'd be interested to hear any other use cases people could imagine. If you want to give this a try, you need an MPX-enabled X server (which you can get from Xorg Jhbuild) and window manager (such as compiz), and you'll need to apply my Vino patches. Enjoy.
Awesome!
Hey! not sure if you read this:
http://blogs.gnome.org/carlosg/2009/01/19/more-multitouchy-fun/
I'm working on improving GTK+ to handle MPX, there are probably some things in the TODO that you'd miss for vino, but getting some feedback would be really great :)
Really cool indeed. I hope to see a future where Gnome/X will be able to mix multiple users (as in posix permissions) in the same session.
Right now you still have to use "shared users". Different "pointers" or users won't get their own bookmarks if they start their webbrowser, for instance.
Just checked out that videos, you must be some really cool guy :)
This video showcases exactly why MPX wins everything. The fact that it's taking a while to be released as stable really helps, too, because so much support is being built in ahead of time.
Chris, you are on my heroes list. (Don't worry, Peter is already there) :)
Remote tech support and collaboration could really rock with this.
I was thinking of extending synergy with a new MPX-capable server, but the code has defeated me more than once.
I second Dylan's comments. I've been a fervent supporter of using multiple cursors in X since 2004 and its absolutely sublime watching MPX taking root. This is a fantastic app for MPX.
Chris,
With regards to building an MPX-enabled X.org server with jhbuild, does that have to replace your existing X server, or can you install it without any conflicts?
Can it, say, be installed in
/opt/mpx?Carlos: Yes, your GDK/XI2 work's exactly what this needs next. ;-) Thanks very much for doing it!
Jeremy: JHBuild isolates your build environment from your main setup in the way you describe. You install into /opt/mpx, or wherever, and can then run the new Xorg inside a "jhbuild shell".
Thanks for the positive feedback, all.
really cool. but you'd have to be careful, when one person logs in as root in a terminal and another quickly uses that terminal ;)
Just an implementation question (I don't know the inner workings of X):
How can you have multiple focused windows? Or is focus related to devices (keyboard, mount etc.) instead of windows?
Thanks.
z0d: You've just described the MPX (multi-pointer X) work -- it moves focus from being a single window per display to a window per pointer/device per display.
Part of the reason it's taken a long time to get MPX ready is that this assumption of a single focus was present in all kinds of other libraries and client toolkits too, so we need to fix those as well.
For good collaborative text editing, look up SubEthaEdit http://www.codingmonkeys.de/subethaedit/
They have solved pretty nicely all kinds of design issues related to multiple users edting the same document.
mox: Sure, but their solution only works for text editing (rather than any application) and required writing all kinds of network synchronization and conflict-resolution algorithms; it's not something average programmers could have done.
The point of my suggestion of MPX as a collaborative framework was to show that such applications wouldn't need to know anything about networking in order to be collaborative, and that makes it a powerful environment for programming.
Hey Chris,
Very interesting, I saw some commercial attempt at something very similar that only tackled multiple pointers. If I can dig out the link I'll send it over.
Its certainly something the real time collaboration space could do with, this is the first implementation I have seen that actually does exactly whats required.
P.S. How have you been, long time no see and all that :)
P.P.S. I may be east coast US at somepoint in the near future with a few days to spare if you want to catch up.
Steven
mox, Chris: Also, SubEthaEdit is OSX only.
Obby is the open spec for concurrent text editing. The stock implementation is the text editor Gobby. Its much akin to SubEthaEdit, but its free & extendable. Ah, sorry, I'm a bit out of date: Obby appears to have been replaced by infinote.
Even with MPX, afaik there are no editors that permit multiple people to work on the same document within a single editor. That would make an interesting project; a Gobby editor built around MPX.
There is however a blog post I'm recalling, a warning to the effect of "dont write a text editor". Indeed, this would be a cute gimmicky feature to prototype out, but writing an actual text editor sounds un-fun. Not unless theres a modularized vim somewhere I can tap.
Is there a possibility to get the patches for vino? The link posted above doesn't seem to work anymore. I get: "cgit error: Invalid request"
Any help would be appreciated!
Thank you
Oops, the URL for the patchset broke in a migration from gitweb to cgit. Here's the current URL:
http://dev.laptop.org/git/users/cjb/vino/log/?h=mpx
how funny, i did almost the same thing you did at almost the same time ;-)
but instead of vino i patched x11vnc:
http://www2.informatik.hu-berlin.de/~beier/code/CollabKit/src/server/x11vnc-0.9.6-multi5.tar.gz
and btw, did you also have problems with XTestFakeDeviceMotionEvent() not sending XI2 events? i had to use xwarppointer() instead...
Woooowwwwwwwwwwww!!!!!! :)
Remote desktop in itself is a great technology..ya but the problem is in getting the hands onto it..seems MPX technology enabled X desktop would fret out the issues..