Chris Ball: Multi-pointer Remote Desktop

Multi-pointer Remote Desktop

Chris Ball — Mon, 26 Jan 2009 05:10:00 +0000

Remote desktop is a pretty useful technology. It would be more useful if it wasn't so competitive, though; when someone joins, the sharer has to sit back and watch, or fight over who gets control. Well, now it's cooperative — here's a video demo:

Download the video (23M)

I should make it clear that I'm responsible only for the smallest piece of this work. Peter Hutterer's MPX is the core technology enabling an X desktop to have more than one pointer and keyboard present on it, and my work was modifying the Vino VNC server to make it understand how MPX works. As shown in the video, this requires no modifications to VNC clients — each client can become another pointer on your X desktop regardless of whether it's running on a Windows, Mac or other Unix machine, which I think makes the whole thing pretty compelling.

My patches against Vino are here, and this is a complete summary of them: for each new client that joins, we create a new keyboard/pointer pair and store its details with that client, then when new events come in for a client we send them through to the appropriate device. That's all. This is still prototype code, so there are bugs, the most noticeable of which is when you have more than one client connected, each client can only see the pointer of themselves and the server, although the server can see everyone's pointer; this will be fixed by having vino_cursor_update_timeout() and parents act on a list of cursor positions rather than a single one. (Ideally, GDK should become aware of multiple pointers, too.)

The next steps are to fix some bugs and get the code upstream into the main Vino repository, so that it's ready for when the distros start shipping a multi-pointer X server — one way to do this would be to have Vino's preferences check to see whether your X server supports MPX, and ask you to choose between sharing your main pointer or granting extra pointers to new clients if it does.

The original motivation for this work, besides it being a useful-sounding idea in general, was as an experiment for a collaboration model for OLPC. A very intuitive way for writing collaborative applications would be to make programs that don't have to understand anything about networking, but do have the idea that there's going to be more than one pointer and keyboard around sometimes and that they should be treated differently, just like when using a games console with multiple controllers.

For example: to write a collaborative text editor using this model, you'd want to color the text written by each participant's keyboard differently, but other than that you don't need to care about any details of networking. (You'd want an out-of-band way for each participant to get a copy of the file they were working on afterwards, though...)

I'd be interested to hear any other use cases people could imagine. If you want to give this a try, you need an MPX-enabled X server (which you can get from Xorg Jhbuild) and window manager (such as compiz), and you'll need to apply my Vino patches. Enjoy.

"Multi-pointer Remote Desktop" by Chrome OS

Sat, 13 Mar 2010 11:17:14 +0000

Chrome OS

"Multi-pointer Remote Desktop" by Compatible Ink Cartridges

Thu, 24 Sep 2009 12:16:15 +0100

Remote desktop in itself is a great technology..ya but the problem is in getting the hands onto it..seems MPX technology enabled X desktop would fret out the issues..

"Multi-pointer Remote Desktop" by Eibriel

Sun, 24 May 2009 21:34:08 +0100

Woooowwwwwwwwwwww!!!!!! :)

"Multi-pointer Remote Desktop" by christian

Mon, 27 Apr 2009 23:31:45 +0100

how funny, i did almost the same thing you did at almost the same time ;-)

but instead of vino i patched x11vnc:

http://www2.informatik.hu-berlin.de/~beier/code/CollabKit/src/server/x11vnc-0.9.6-multi5.tar.gz

and btw, did you also have problems with XTestFakeDeviceMotionEvent() not sending XI2 events? i had to use xwarppointer() instead...

"Multi-pointer Remote Desktop" by Chris

Mon, 16 Mar 2009 21:59:50 +0000

Oops, the URL for the patchset broke in a migration from gitweb to cgit. Here's the current URL:

http://dev.laptop.org/git/users/cjb/vino/log/?h=mpx

"Multi-pointer Remote Desktop" by Siegfried Pammer

Mon, 16 Mar 2009 14:07:41 +0000

Is there a possibility to get the patches for vino? The link posted above doesn't seem to work anymore. I get: "cgit error: Invalid request"

Any help would be appreciated!

Thank you

"Multi-pointer Remote Desktop" by rektide

Wed, 28 Jan 2009 17:50:26 +0000

Obby is the open spec for concurrent text editing. The stock implementation is the text editor Gobby. Its much akin to SubEthaEdit, but its free & extendable. Ah, sorry, I'm a bit out of date: Obby appears to have been replaced by infinote.

Even with MPX, afaik there are no editors that permit multiple people to work on the same document within a single editor. That would make an interesting project; a Gobby editor built around MPX.

There is however a blog post I'm recalling, a warning to the effect of "dont write a text editor". Indeed, this would be a cute gimmicky feature to prototype out, but writing an actual text editor sounds un-fun. Not unless theres a modularized vim somewhere I can tap.

"Multi-pointer Remote Desktop" by z0d

Wed, 28 Jan 2009 13:31:17 +0000

mox, Chris: Also, SubEthaEdit is OSX only.

"Multi-pointer Remote Desktop" by Steven

Tue, 27 Jan 2009 22:14:40 +0000

Hey Chris,

Very interesting, I saw some commercial attempt at something very similar that only tackled multiple pointers. If I can dig out the link I'll send it over.

Its certainly something the real time collaboration space could do with, this is the first implementation I have seen that actually does exactly whats required.

P.S. How have you been, long time no see and all that :)

P.P.S. I may be east coast US at somepoint in the near future with a few days to spare if you want to catch up.

Steven

"Multi-pointer Remote Desktop" by Chris

Tue, 27 Jan 2009 19:46:10 +0000

mox: Sure, but their solution only works for text editing (rather than any application) and required writing all kinds of network synchronization and conflict-resolution algorithms; it's not something average programmers could have done.

The point of my suggestion of MPX as a collaborative framework was to show that such applications wouldn't need to know anything about networking in order to be collaborative, and that makes it a powerful environment for programming.

"Multi-pointer Remote Desktop" by mox

Tue, 27 Jan 2009 19:39:08 +0000

For good collaborative text editing, look up SubEthaEdit http://www.codingmonkeys.de/subethaedit/

They have solved pretty nicely all kinds of design issues related to multiple users edting the same document.

"Multi-pointer Remote Desktop" by Chris

Tue, 27 Jan 2009 15:58:36 +0000

z0d: You've just described the MPX (multi-pointer X) work -- it moves focus from being a single window per display to a window per pointer/device per display.

Part of the reason it's taken a long time to get MPX ready is that this assumption of a single focus was present in all kinds of other libraries and client toolkits too, so we need to fix those as well.

"Multi-pointer Remote Desktop" by z0d

Tue, 27 Jan 2009 14:44:18 +0000

Just an implementation question (I don't know the inner workings of X):

How can you have multiple focused windows? Or is focus related to devices (keyboard, mount etc.) instead of windows?

Thanks.

"Multi-pointer Remote Desktop" by baze

Tue, 27 Jan 2009 13:22:45 +0000

really cool. but you'd have to be careful, when one person logs in as root in a terminal and another quickly uses that terminal ;)

"Multi-pointer Remote Desktop" by Chris

Tue, 27 Jan 2009 05:26:49 +0000

Carlos: Yes, your GDK/XI2 work's exactly what this needs next. ;-) Thanks very much for doing it!

Jeremy: JHBuild isolates your build environment from your main setup in the way you describe. You install into /opt/mpx, or wherever, and can then run the new Xorg inside a "jhbuild shell".

Thanks for the positive feedback, all.

"Multi-pointer Remote Desktop" by Jeremy

Tue, 27 Jan 2009 04:42:50 +0000

Chris,

With regards to building an MPX-enabled X.org server with jhbuild, does that have to replace your existing X server, or can you install it without any conflicts?

Can it, say, be installed in /opt/mpx?

"Multi-pointer Remote Desktop" by rektide

Mon, 26 Jan 2009 21:53:43 +0000

I was thinking of extending synergy with a new MPX-capable server, but the code has defeated me more than once.

I second Dylan's comments. I've been a fervent supporter of using multiple cursors in X since 2004 and its absolutely sublime watching MPX taking root. This is a fantastic app for MPX.

"Multi-pointer Remote Desktop" by Donnie Berkholz

Mon, 26 Jan 2009 17:40:53 +0000

Remote tech support and collaboration could really rock with this.

"Multi-pointer Remote Desktop" by Dylan McCall

Mon, 26 Jan 2009 15:21:24 +0000

This video showcases exactly why MPX wins everything. The fact that it's taking a while to be released as stable really helps, too, because so much support is being built in ahead of time.

Chris, you are on my heroes list. (Don't worry, Peter is already there) :)

"Multi-pointer Remote Desktop" by ikke

Mon, 26 Jan 2009 15:05:21 +0000

Just checked out that videos, you must be some really cool guy :)

"Multi-pointer Remote Desktop" by nona

Mon, 26 Jan 2009 12:37:33 +0000

Really cool indeed. I hope to see a future where Gnome/X will be able to mix multiple users (as in posix permissions) in the same session.

Right now you still have to use "shared users". Different "pointers" or users won't get their own bookmarks if they start their webbrowser, for instance.

"Multi-pointer Remote Desktop" by Carlos Garnacho

Mon, 26 Jan 2009 12:14:30 +0000

Hey! not sure if you read this:

http://blogs.gnome.org/carlosg/2009/01/19/more-multitouchy-fun/

I'm working on improving GTK+ to handle MPX, there are probably some things in the TODO that you'd miss for vino, but getting some feedback would be really great :)

"Multi-pointer Remote Desktop" by Marius Gedminas

Mon, 26 Jan 2009 07:55:39 +0000

Awesome!