Multi-pointer Remote Desktop —

Remote desktop is a pretty useful technology. It would be more useful if it wasn’t so competitive, though; when someone joins, the sharer has to sit back and watch, or fight over who gets control. Well, now it’s cooperative — here’s a video demo:



Download the video (23M)

I should make it clear that I’m responsible only for the smallest piece of this work. Peter Hutterer’s MPX is the core technology enabling an X desktop to have more than one pointer and keyboard present on it, and my work was modifying the Vino VNC server to make it understand how MPX works. As shown in the video, this requires no modifications to VNC clients — each client can become another pointer on your X desktop regardless of whether it’s running on a Windows, Mac or other Unix machine, which I think makes the whole thing pretty compelling.

My patches against Vino are here, and this is a complete summary of them: for each new client that joins, we create a new keyboard/pointer pair and store its details with that client, then when new events come in for a client we send them through to the appropriate device. That’s all. This is still prototype code, so there are bugs, the most noticeable of which is when you have more than one client connected, each client can only see the pointer of themselves and the server, although the server can see everyone’s pointer; this will be fixed by having vino_cursor_update_timeout() and parents act on a list of cursor positions rather than a single one. (Ideally, GDK should become aware of multiple pointers, too.)

The next steps are to fix some bugs and get the code upstream into the main Vino repository, so that it’s ready for when the distros start shipping a multi-pointer X server — one way to do this would be to have Vino’s preferences check to see whether your X server supports MPX, and ask you to choose between sharing your main pointer or granting extra pointers to new clients if it does.

The original motivation for this work, besides it being a useful-sounding idea in general, was as an experiment for a collaboration model for OLPC. A very intuitive way for writing collaborative applications would be to make programs that don’t have to understand anything about networking, but do have the idea that there’s going to be more than one pointer and keyboard around sometimes and that they should be treated differently, just like when using a games console with multiple controllers.

For example: to write a collaborative text editor using this model, you’d want to color the text written by each participant’s keyboard differently, but other than that you don’t need to care about any details of networking. (You’d want an out-of-band way for each participant to get a copy of the file they were working on afterwards, though…)

I’d be interested to hear any other use cases people could imagine. If you want to give this a try, you need an MPX-enabled X server (which you can get from Xorg Jhbuild) and window manager (such as compiz), and you’ll need to apply my Vino patches. Enjoy.

Comments

  3. Really cool indeed. I hope to see a future where Gnome/X will be able to mix multiple users (as in posix permissions) in the same session.

    Right now you still have to use “shared users”. Different “pointers” or users won’t get their own bookmarks if they start their webbrowser, for instance.

    Reply ↓

  5. This video showcases exactly why MPX wins everything. The fact that it’s taking a while to be released as stable really helps, too, because so much support is being built in ahead of time.

    Chris, you are on my heroes list. (Don’t worry, Peter is already there) 🙂

    Reply ↓

  7. I was thinking of extending synergy with a new MPX-capable server, but the code has defeated me more than once.

    I second Dylan’s comments. I’ve been a fervent supporter of using multiple cursors in X since 2004 and its absolutely sublime watching MPX taking root. This is a **fantastic** app for MPX.

    Reply ↓

  8. Chris,

    With regards to building an MPX-enabled X.org server with jhbuild, does that have to replace your existing X server, or can you install it without any conflicts?

    Can it, say, be installed in /opt/mpx?

    Reply ↓

  9. Carlos:
    Yes, your GDK/XI2 work’s exactly what this needs next. 😉 Thanks very much for doing it!

    Jeremy:
    JHBuild isolates your build environment from your main setup in the way you describe. You install into /opt/mpx, or wherever, and can then run the new Xorg inside a “jhbuild shell”.

    Thanks for the positive feedback, all.

    Reply ↓

  10. really cool.
    but you’d have to be careful, when one person logs in as root in a terminal and another quickly uses that terminal 😉

    Reply ↓

  11. Just an implementation question (I don’t know the inner workings of X):

    How can you have multiple focused windows? Or is focus related to devices (keyboard, mount etc.) instead of windows?

    Thanks.

    Reply ↓

  12. z0d: You’ve just described the MPX (multi-pointer X) work — it moves focus from being a single window per display to a window per pointer/device per display.

    Part of the reason it’s taken a long time to get MPX ready is that this assumption of a single focus was present in all kinds of other libraries and client toolkits too, so we need to fix those as well.

    Reply ↓

  14. mox: Sure, but their solution only works for text editing (rather than any application) and required writing all kinds of network synchronization and conflict-resolution algorithms; it’s not something average programmers could have done.

    The point of my suggestion of MPX as a collaborative framework was to show that such applications wouldn’t need to know anything about networking in order to be collaborative, and that makes it a powerful environment for programming.

    Reply ↓

  15. Hey Chris,

    Very interesting, I saw some commercial attempt at something very similar that only tackled multiple pointers. If I can dig out the link I’ll send it over.

    Its certainly something the real time collaboration space could do with, this is the first implementation I have seen that actually does exactly whats required.

    P.S. How have you been, long time no see and all that 🙂

    P.P.S. I may be east coast US at somepoint in the near future with a few days to spare if you want to catch up.

    Steven

    Reply ↓

  17. Obby is the open spec for concurrent text editing. The stock implementation is the text editor Gobby. Its much akin to SubEthaEdit, but its free & extendable. Ah, sorry, I’m a bit out of date: Obby appears to have been replaced by [infinote](http://gobby.0x539.de/trac/wiki/Infinote/Protocol).

    Even with MPX, afaik there are no editors that permit multiple people to work on the same document within a single editor. That would make an interesting project; a Gobby editor built around MPX.

    There is however a blog post I’m recalling, a warning to the effect of “dont write a text editor”. Indeed, this would be a cute gimmicky feature to prototype out, but writing an actual text editor sounds un-fun. Not unless theres a modularized vim somewhere I can tap.

    Reply ↓

  18. Is there a possibility to get the patches for vino? The link posted above doesn’t seem to work anymore. I get: “cgit error: Invalid request”

    Any help would be appreciated!

    Thank you

    Reply ↓

  22. I have an application that would work well this type of setup. In fact, I’ve been looking for software like this.

    The application is an RPG module for Vassal game engine. I have used a Vassal card deck to implement a time queue for time/phase order events. Players have private hands where they click on cards to take actions and place the events into the queue. In addition to laptops for each player/GM there is playboard “machine”. I have 2 24″ monitors stacked and then placed horizontal rather than vertical. This gives a playing surface of about 2’x 2′. Tokens on this board would be handled by all players.

    Now Vassal will duplicate the playboard on each machine, so for traditional games players are just playing through the net. But in this case, it is the convenience of having a shared board and the lack of screen real estate on the laptops that would make this software essential to how I would to have the game played.

    Reply ↓

  23. I tried to install this patch on Ubuntu Maverick, which ships with a MPX aware X-Server. Multi-pointer in general works, but the patched vino does not. I works well, but mouse and keyboard are competitive. Anyone got this running on Ubuntu Maverick? Any ideas how to make this work?

    Reply ↓

  24. I’ve just run into this and I’m interested how did this turn out? It’s 10 years later and it still seems there is no native solution.

    Reply ↓

Leave a Reply

Your email address will not be published. Required fields are marked *