Building a platform

Posted by on Jun 13, 2015 in KDE, ownCloud | 12 Comments

From the very beginning ownCloud has had bigger ambitions then just being a file sync and share tool. This is apparent from the name ownCloud. Today, we have more than our documents and photos online. Our social networks and shared thoughts, our appointments and shopping lists, audio and video conversations all happen and are stored somehwere ‘in the cloud’, all connected. You can comment on a song you like for others to see or share an appointment with co workers. ownCloud means to give you a chance to bring all that back under your control!

But even with the scalability of the open source development model, ownCloud could not cover the almost infinite number of things happening online. Rather than creating one huge application which does it all, ownCloud was designed to have a relatively small core and implement most of its functionality in independent apps. Examples include very central functionality like sharing, versioning, the trash bin, encryption and LDAP support.

Such a modular architecture makes complex software like ownCloud more manageable and easier to develop and maintain. But we also introduced an interface for admins to manage the apps of an ownCloud instance.

We did that because ownCloud was meant to be a platform for third party applications. Sync and share is absolutely the core feature of ownCloud but we always wanted to build something more. A platform with an ecosystem where other developers can build applications on top.

Examples for apps are:
– New ways to authenticate users like LDAP
– New connectors to storage backends like FTP or Dropbox
– New file handling features like versioning or encryption
– Features like Calendar, Contacts, Mail, Music and so on.
– Completely new features like RSS readers, Document editing, chat or password management.

Growing the app ecosystem

At the moment a default ownCloud server installation already includes a handful of apps that provide basic functionality. Additionally around 50 third party apps are available to install to get more functionality. This is really great. But let´s think bigger here.

If you look at other platforms like Windows, Mac, iOS, Android or Drupal, KDE and GNOME then you see that there are hundreds to hundreds of thousands of independent apps available to run on top. Saying that we could reach the same number in the short term would be unrealistic. But we definitely want to have a lot more than we have today and become a real platform.

What do we have to do in ownCloud to grow from the current 50 to let´s say 1000 apps in the near future? For that we need to follow three important principles in developing ownCloud related to mindset, process and structure: supporting decentralization, building a great technology platform and having clear user communication and promotion.

Supporting Decentralization

If we are to build an ecosystem, we have to accept that we will not have everybody who works on ownCloud in one place. Not every app will be on github, not every app developer will use our testing tools or coding guidelines, will be on our mailing list or can ask questions in person. Key elements here are our infrastructure and communication.

Infrastructure

Not every app will be developed on Github or even the ownCloud Github organization. Our github repo is open for everyone who wants to developer their app close to the core code and team to benefit from help and code reviews from other people. Anybody can mail our developer list to request a new repository. But they don’t have to – apps can be developed anywhere and we have to adjust our processes to this.

For example, even if we’d update all applications in the ownCloud github organization to an API change, we can’t deprecate the API! We don’t know and can’t control who else depends on this interface and if we do not take this bigger view, we won’t reach the thousands of third party apps we want.

This also means we have to deal with third party apps not using our translation system, unit tests, coding guidelines or anything else part of our infrastructure! Our API’s and the way we design ownCloud has to fit with any way of working.

Communication

The second element to supporting a decentralized ownCloud ecosystem is our communication. We will have to realize that the core team can’t know every third party developer in person. It is very useful to be able to talk to the core developer of an app to discuss an API or UI change or a possible bug. This open communication is one of the core benefits of an open development process. But it is pretty clear that this doesn’t scale if you have > 1000 developers and apps. So every attempt to centralize the communication or base it on personal contact or friendship alone will fail or limit the growth of ownCloud. So every communication of APIs, best coding practices, release schedules or any other information that effect app developers has to happen in a way that works for a big ecosystem.

This means there is a challenge with getting announcements to third party developers. Not all of them will be on our mailing-lists or IRC channels or will attend our meetings or conferences. You might say that it is enough to discuss topics that effect app developers on this channels or events. But we have to realize that not everyone can be a core member of the ownCloud contributor community to follow this channels. All relevant information and announcements has to be made available to all app developers via our documentation, blog posts or other scalable ways of communication.

Building a Great Technology Platform

The first step to making ownCloud a platform is technology. We need to offer a well documented, stable and complete base for third party developers to build on.

Stable APIs

It is key here to realize that not everybody has as much time to spend on ownCloud as core developers do. Updating an application because one of the APIs it uses has been changed is not only a chore without bringing any user benefits, it can take a long time or never happen if an app is not actively maintained.

Platforms like Windows, Mac, Android and so on provide a stable API that is backward compatible for years. We have to do the same if we want to be a useful platform and grow a big ecosystem. iOS and Android apps from 5 years ago still work with the latest release. Windows and Mac software from over 10 years ago works with the latest releases. This has to be our benchmark to reach the same level of stability.

And stable APIs doesn’t only mean stable PHP API. It is the same for our JS code or other changes that we might do in CSS, icons or other areas that might effect how third party apps work. If we do a change in the ownCloud core that breaks an older third party app then this is our fault and not the responsibility of the third party developer to constantly adapt the app to the core API changes. Our code checker is a good way to make sure that no private APIs are used that might break the app in an upcoming release. But not everything can be checked automatically. We have to keep this in mind when we develop the core and the APIs.

When we have to make changes to the API, it is important to announce them in advance and keep the ‘old’ API still available rather than entirely replacing it. A new API should thus be introduced alongside the old API, which is then deprecated.

I see three years as the sweet spot before we can remove a deprecated API, so developers and users have enough time to migrate. Of course, security can be an exception after careful considerations. It can also be helpful to create tools that help developers migrate to a new API.

Of course, apps which use private APIs of ownCloud will simply have to migrate to public API. If a developer needs to use private APIs, we have to create a public API for him or her to use! App developers can discuss API needs on our development list or in github.

Documentation

Not all developers are deeply into ownCloud code and super experience developing apps. To aid app developers, our APIs need to be designed to be simple to use. Developer documentation is also extremely important. We can’t expect that third party developers read code from core or other apps to learn how a specific API has to be used or how something has to be implemented. So we have to provide high level developer documentation. Additionally tutorials, example apps and all kind of other resources help developers get up and running.

User Communication and Promotion

The third main principle needed to build a large app ecosystem is communication. We need to explain to both developers and users that we’re serious about apps, how they can get or develop apps and where to find help. We have to promote apps and manage expectations.

Promote apps

Third party developers need our help to promote their apps. We have to talk about them and blog about them to make them successful. Every ownCloud third party app should have the visibility that it deserves. One thing that ownCloud can do to help here is our app marketplace / app store at apps.owncloud.com. This makes it easy for third part developers to publish their app and make them available to all ownCloud users. The more popular these third party apps are the better for everyone in ownCloud.

Manage expectations

Not every third party app has the same level of quality, security and user experience. Instead of just rejecting low quality apps we have to explain to our users what to expect from which app. This also helps the developer of third party app developers to improve their apps to reach a higher quality level.

These levels are partly reflected in the user ratings from the app store. But additionally we want to introduce different quality levels for apps which we actively triage. Apps have to comply to certain quality qoals to to be on a specific quality level. We will announce the changes that we introduce in ownCloud 8.1 here soon.

Conclusion

The key to building an ecosystem is taking the needs and requirements of third part developers seriously. If the development documentation is not good, if an API is missing or broken, if a questions from a third party developer is not answered fast enough then we as core developers have failed. It is the responsibility of all of us to always remember that.

 

 

12 Comments

  1. tiagoprn
    13/06/2015

    It was pretty good reading about all that care and plans for the platform. I have a droplet on DigitalOcean where I primarily use the sync feature, and would line to help grow the ecossystem as a developer. But the barrier here is that I am a python developer, and as such it is not clear to me if the core owncloud functionalities are exposed through a REST API that I could use to developer my own apps or customize another (e.g. the Calendar, which is my second interest). The last time I checked there was an API on the “enterprise” edition only, which I think is too much limiting for a platform with such openess claims. To me, that seems critical if the project has ambitions for growing… if you could extend an invitation and market yourself to other developer comunities like python, ruby and others, I think than owncloud could reach its full potential as an open platform.

    Reply
    • Frank Karlitschek
      14/06/2015

      @tiagoprn All APIs are full open source. I’m not sure which API you are referring to specifically. But if you mean to provisioning API that can be used to create, change and delete users then please look try again ownCloud 8.0 or 8.1
      This API was open sourced last year.
      Let me know if you have further questions.

      Reply
  2. sebas
    14/06/2015

    Do your “platformization” plans also include the Qt-based owncloud client?

    As you perhaps remember, I’ve worked on a client a while ago which was based on this code. Its goal was to create a client with a converging user interface that integrates really well with the Plasma workspace. For that, I’ve used a library included in the owncloud client, and cleaned that up a bit, mainly for better separation of logic and presentation. From that point on, I could use the meat of the existing owncloud client to base my own user interface on, benefit from the low-level work you guys do and concentrate my efforts on the bits that make a difference to the user experience. It’s not meant as a competitor, but rather as a client for a different use case, deeply integrated and also beyond the desktop. I got it to work pretty well at some point, but I wasn’t able to keep up with changes in the owncloud client. I managed to put some work in in the meantime, but never gotten it back to a quality level I could actually announce or ship.

    In part, that’s because the owncloud client develops at a rather fast pace (not a bad thing, really). A part of the problem was also that the library was never much more than an internal implementation detail, without any promises about API stability. Now I was able to employ the “personal communication” channels you talk about, but in the end, my available time did not allow me to keep up, so for now the code is bitrotting.

    Knowing that your “platform promises” (or at least plans, I don’t need a 100%-for-sure-or-else promise) also include the Qt-based owncloud client library, I think it’d be interesting to pick up that work again, and maybe have it come to fruition finally.

    I’d love to see a well-integrated synching solution for Plasma, and I think there’s a great deal of overlap in owncloud’s and Plasma’s goals regarding that (not surprising, given its pedigree), so maybe we can team up and make it happen?

    Reply
    • Frank Karlitschek
      14/06/2015

      Hi Sebas.

      In my post I referred mainly to server side apps and extensions. But you are right. This should be the same for clients and stable APIs of the client library.

      Maybe we should start a new discussion between you, Klaas and me what we can do to have a stable library that does what you need. What do you think?

      Frank

      Reply
      • sebas
        14/06/2015

        Sounds great, maybe we can have this in person over a nice drink or dinner at Akademy? Let’s talk about that via email.

        Reply
  3. Pascal d'Hermilly
    15/06/2015

    It seems to me that a part of Owncloud’s frontend technology is old.
    Why does it have to do a full change of the webpage when changing apps from eg. Files to Contacts?
    That makes it slower and unusable for e.g. music streaming – the music stops each time you navigate. (Although I’ve had to give up using the music streaming since the music app simply doesn’t scale to bigger music collections – it creates a big block in the app for each album and since all albums are always showed it quickly amounts to gigabytes of memory use.)

    Reply
  4. joe
    17/06/2015

    Apple’s iPhone was not a Platform when it first started out. They first of all created some great apps on their own, showed it was possible, and only then started opening up for outside apps. By that time, people saw the potential and the high quality. I would much prefer a focus on building strong and bug-free core apps (files, calendar, contacts, documents) than a focus on being something for everyone.

    Reply
    • kuba
      17/06/2015

      Indeed.

      Reply
    • Francis Irving
      20/10/2015

      Agreed!

      My experience trying OwnCloud in detail a couple of years ago, and playing with the demo again just now, is that those core apps are not yet good enough.

      I’d start using it if they were delightful.

      I think though that OwnCloud is quite targeted at businesses, which might have different needs from me as a consumer.

      Reply
  5. ownCloud 8.1 – Raising the Bar on Security and Performance | ownCloud.org
    07/07/2015

    […] ownCloud is meant to be a platform, much attention went into improving the ownCloud API. Over 190 new functions became available while […]

    Reply
  6. ownCloud 8.1 Brings Massively Improved API for Developers | ownCloud.org
    10/07/2015

    […] when an API was introduced as well as when they get removed. This bring us closer to our goal of building a platform for application developers with ownCloud. Going forward, our goal is to keep API’s working for three years after we have […]

    Reply

Leave a Reply