General Posts

Looking Toward the Horizon of Mobile Computing

It is not saying anything new to point out that the iPhone changed the way we thought about and interacted with smart-phones. In the years following the Android phones brought us a vision of what a cloud-centric phone OS on many vendors hardware could provide if it was well supported. And the Palm webOS phones showed us what intuitive multi-tasking and a great notification system could provide.

With the improvements to the hardware and software over the two following years of iPhones and their respective software we got most the features we could want. I assume some sort of multi-tasking is coming in the 4th iteration. So we have reached a place where the revolution has evolved to a mature place of mobile-specialized OS’s and great hardware.

But where is the next revolution? I don’t think it is in the tablet space, which is being heralded as the next big consumer computing cash-cow. I think at least the next big evolution, or progression would be context-aware computing and inter-computer interaction.

What I mean by that is that I have massive headaches in some of the artifical boundaries, and poorly thought-out aspects of how our phones/mp3 players, tablets/netbooks, and regular laptops and desktops interact with eachother.

In order to explain this, let’s focus just on phones and go through the situation we have today.

I have drawn this little diagram that highlights what is considered basically the best way to ‘sync’ data on your phone.

The basics are that the best way to sync small data – email, contacts, calendar, and carrier data comes from the ‘cloud.’ The cloud here being any internet-based or telephony-based data that lives on a server somewhere far away and is ubiquitously available via a link to the net.

That same data is accessible via web services or the same as mobile protocols to your PC.

But when it comes to larger data, files, music, video, pictures, and some larger data sets from your app, the best way to move it between the PC and other computing devices is to connect a cable, and utilize the computer program specific to the device you are syncing. Every phone as a different non-standard way to handle this.

What works with this?

  • iTunes lacks features for power users, but it gets the syncing job done well enough (finally as of iTunes 8 which gives you the ability to change the ‘media type’ from music to audiobook/podcast/etc).
  • Cloud syncing is GREAT for information that best lives in cloud (i.e. Email, contacts, calendars)
  • Phones in general have gotten pretty good for playing music, making calls and passing text messages, and GPS.
  • Computers are great at creating and storing files, documents, and media.

What Doesn’t Work

  • iTunes is ‘Apple Only’ and it isn’t flexible. Ever want to sync the last 6 unplayed episodes of one podcast, but keep all the files for another podcast. Can’t do it with iTunes without serious playlist weirdness. Plus, it is difficult to get bi-directional syncing. Unless Apple decides to make special arrangements (ala voice memos) you can’t create or download media to the device and have it sync back to the computer easily.
  • Cloud syncing isn’t currently practical for larger media. It may get there, but our media libraries, and local storage capabilities (in both mobile devices and regular hard drives) are out growing our broadband speeds in the US. Plus, even if our broadband sped up a lot, it would be hard to match local network speeds or direct cable syncing.
  • The current protocol that makes cloud syncing a semi-decent user experience is proprietary and your service of choice (ala Gmail for example) has to pay Microsoft and then implement the protocol. And the protocol isn’t extensible or adaptable. This leaves us with a fragmented messaging experience. Multiple apps to handle chat, SMS/MMS, Email, etc – where we would benefit from commonality.
  • There is no standard way to sync contact info between devices and web apps – think about the facebook integration on the iphone app for facebook, or the direct built-in facebook capabilities in webOS and Android – so that when you update your info on facebook it goes to your friends directly.
  • So if you think about use-cases my phone usage based on what does and doesn’t work, and a lot of other people I know too, works out about like this: If I’m out and about, I get my email, set appointments in my calendar, create new contacts, and check and update facebook and twitter. This is of course in addition to taking calls. But the minute I sit down at a PC, I do all that stuff on the PC – because it is just easier to edit everything (except maybe twitter) on a PC. Especially longer emails and things that benefit from more typing. But the only reason I pick up my phone is to take a call or answer a text. And it is kinda a pain in the ass once I’m settled at a PC if my phone is sitting somewhere a ways away.

    So, here is my list of thoughts on how we might address some of those things that doesn’t work and maybe improve the user experience.

    • Let’s connect our phones to our PCs more so we can pass information more easily between the two.
      • Start with the phone calls – let’s pass the phone calls to the computer and use the computers mic and speakers.
        • The simple answer is have the computer support a hands-free profile, and the phone would connect to it like an earpiece. Then some type of notification would have to come up when there was a call.
        • The more complex answer would be to run a VOIP or some type of messaging client on the PC, and the phone would receive the call, and forward it as a VOIP call to the PC.
      • Step 2 is to expose a common driver/API architecture to expose onboard features of the phone or devices on the computer to eachother. So when the phone connects to the PC via USB or Bluetooth, let the computer see a USB hub with a camera, GPS, mic/speakers/audio device, etc. (See table below for a running tally of what I’m thinking)
      • Computer sees Phones as:

        Device Via
        USB Hub Driver / virtualization layer?
        Bluetooth adapter Driver / BT profile (a pass-through/relay type profile)
        External Audio Device Driver
        Touchpad Mouse / Keyboard BT Profile / driver via USB
        Camera Driver
        GPS Driver
        External Storage Device/SD Card reader Driver and network sharing
        3G Modem Driver / Bluetooth PAN
        External Display (windows-sideshow style) Driver (or app maybe?)
        Voice Gateway – Expose the GSM modem or VOIP gateway Driver

        Phone sees computer as:

        Device Via
        External Storage Network Share / special app
        Net connection Driver / Bluetooth PAN
        USB/Bluetooth Keyboard Driver/BT profile/Server App
        External Audio device w/ mic and speakers Driver
        USB Hub – the current apple ability to have peripherals added to the phone is vastly under-utilized Driver / Extention to the BT profiles to encompass Wireless USB (since wireless USB is on the way out)
        Video / Webcam / still cam Driver
        Secondary CPU / Processor core and GPU Virtualization layer

      • The next level would be at the application and OS.
        • Seamlessly move apps from the phone to the PC so you can interact with them from the PC. This way you can pick-up where you left off.
          • Allow a framework or API to sync applications on connection of PC to computer.
            • An example – listening to music on the way to work, get to your desk, phone prompts “Do you want to transfer this to the PC app, move the App to the PC, or continue on the phone?”
              • Move to PC app – music pauses on phone, music app (iTunes, winamp, etc) opens, and starts playing the same file off the phone from where it was paused.
              • Move app to phone – see diagram
              • Keep on phone – nothing changes
          • The final step would be having the apps be aware of the computer connection. Similar to the ‘app syncing, but instead the phone app could be transferred to the PC, but when it got there it would have a specific UI for that situation. It would see the CPU on the PC as a resource available to it, and start drawing a more desktop/laptop-centric UI with some expanded feature set. That way the transition to the PC would produce an experience tailored to the PC and have a seamless transition. This would have to have some type of virtualization layer that exposes the CPU and possibly memory or GPU exposed to the phone OS in a way that it can use.
            • This would allow for more ‘in-between’ situations to be handled with a middle UI for an app. So an iPad could have a version of the UI, the computer another, and the phone another – but all do similar tasks.
          • Moving away from the PC-phone interaction and on to messaging. I’m envisioning a new messaging hub. A timeline where you can see all your calls, emails, texts, chat messages, and whatever else you want. All this information pushed to you (instead of polling – which is so 1999). While the implementation details don’t have to be particular on this protocol, it needs to be open, and extensible (that is why I keep thinking about XMPP for the task, like what Wave, Gchat, and facebook chat are based upon). This protocol should have a built-in way to keep track of a message ‘thread’ so that no matter what medium you use, you can see how the chain of messages connected. This protocol should also support calendar events / invites as part of its payload, and it should have some way of a person updating their contact information remotely and if you are ‘subscribed’ to it, you would get the update (if they allowed you to, it would have to be a two-way subscription – with fine-grained controls for this)
            • This would need to have enterprise management features – as in the ability to sync a global contact lists, and manage the security policies of who can see what or book resources.
            • Bonus points for the protocol being aware of when a service like Google Voice forwards you a text message as an email, so you don’t double-up on notifications.
            • This would be a great way to handle generic notifications – meaning if it is extensible it could work for facebook updates, twitter posts, blog/RSS notification if the user so chose to add those features.
          • The final point that I think we should address in the PC-to-Phone interaction is syncing. We need a better, and flexible way of syncing.
            • I believe that as a starting base-point or guideline is that the phone OS makers should pick a single folder in the file-system, and make that a ‘shared space.’ Meaning apps can install to a program location, but they should store their data in the shared space. They should enable file sharing over the network, and the ability for the phone to act as an external storage device (so the phone appears as if it is a USB hard drive). This folder should be appear as a shared folder over the network, and should appear as the root folder when the phone is connected.
              • Some basic common sub-folders within this folder should be agreed upon. I’m thinking:
                • /media
                  • /media/video
                  • /media/music
                  • /media/podcast
                • /documents
                • /app data
              • A matching folder structure should be tied to the computer for backup and ease-of-sync purposes.
            • Secondly I think that having a ‘dropbox’ like tool running on any approved PC, and the phone, which watches for changes to the files in the shared folders. If the PC is in range or connected via USB/Bluetooth this tool would sync the file changes. I think this is particularly important for the documents that can be easily edited, but it gives you a way to back up pictures and videos easily also which is a plus.
            • Finally an agreed upon transaction model for this syncing for libraries. I just mean that some file types, or apps build up a collection of metadata on files, and call that a library. It is fancy way of saying that an application has an internal database of the files and the info about the files that are important to it. iTunes, the music apps on the phones, winamp, songbird, iPhoto, etc.
              • Let’s standardize the way changes are communicated to be bidirectional. This could be a part of the tool above, but if a change happens to a file, or a file is added to the library, when the phone and the PC connect either one can say “Hey, I’ve added files or modified files – do you want the update”.
                • I don’t care what the ‘database’ is that is tied to the app managing your media files, but the way in which the changes are communicated so that a transaction (file updates, etc) take place need to be standardized. By not necessitating a specific back-end to this process we can keep our current DRM schemes I believe for music and video.
              • Also, let’s standardize a smart-playlist-like functionality for syncing collections like podcasts, audiobooks, and series of talks or lectures. So that you can individually say this file, or this series, or this collection is to be synced. If it changes, sync it to wherever doesn’t have the changes, and if there is a conflict – modified in both places between syncs, ask me what to do.
              • The key is, that this has to be seamless for the users – which is why I like dropbox, it is always running and as soon as I save a file it is watching it uploads it to my cloud storage. This could be like that, only just between the PC and the phone and could handle libraries built out of links to files and metadata. (you could add the cloud component too, but a more standard way of handling this would be good)

              So, there you have it, my cell-phone OS wishlist. This is true of the new Windows Phone 7 Series (someday I believe Microsoft will get this naming thing down so it isn’t so silly all the time) which I just heard about 10 minutes ago. Nothing in there suggests to me that Microsoft is doing anything in this direction beyond their typical exchange server support.

              Now, my question to you smart-phone users out there; What is your beef with how your PC and phone interact? What would improve that? And, would any of this help you use a phone or a tablet in a more effective and satisfying way? Or is it just me who doesn’t like the way things are now?