Participate
| You must be registered and signed in to vote and post content. | |
 |
 |
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
| |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
I'm sure many of you will have experienced the problems of playing multimedia on phones - much of your content won't play, even though the phone apparently supports the format you are trying to use. There are many reasons for this - e.g. bit rates for even SD content are too high for phones. However, another key issue is that testing multimedia content is hard - you can test multimedia , demuxers etc individually to death, and still have problems, due to the way things work at a system level (e.g. a bad encoding - not the phone's fault, but still disappointing for the user and not an excuse with service providers). The solution is to test with as much content as possible, but this takes time. Fortunately, this problem seems ideal for a collaborative community, and so I propose a playability database. Symbian would host the database, and copyright-free (or contributed license-free) content. The database would list use-cases - how to run the use-case, the platform version etc. People contributing testing time would run a use-case, and report it either working, or if it fails, open a bug and link the test to the bugzilla entry. The database would show the least recently run test first, thus optimising coverage. This database shares the cost of testing the platform amongst the community in a way not open to rivals, and helps give Symbian a competitive advantage in playability. What do you think? 
Posted on 09/18/2009 03:24 PM BST
, Last Modified on 08/12/2010 10:32 AM BST
 Comments (17)
Hi Martin, This is a really good idea. I think a similar thing should be good for other 'hard to automate' test areas (UI, graphics etc.) An important part of this will be the ease of access to a working version of the platform. A full PDK download might be a bit unwieldy for some people, so we should give them access to a small, ready to run version of the system. This could take the form of a stripped down epoc32 or better yet, a ROM image (when these become available).
1
 
0
I support this because I think that the creation of such a database would be valuable to the whole industry and we're well placed to do it. However, I have some reservations about the validity of sharing playability results, simply because different devices from different manufacturers will have different components (such as codecs) which are essential parts of the system which determines the result. Not saying that they shouldn't be shared, just that the database probably has to account for the specific hardware and software combinations involved, rather than just the platform version - this makes the testing space even larger. I hope that on the plus side, this will encourage codec vendors to engage with the platform further and pre-validate their codecs against (at least some of) our tests with our hardware reference execution environments.
0
0
>> different devices from different manufacturers will have different components (such as codecs) which are essential parts of the system which determines the result This is a valid comment, but to add some more details, I see there are two main types of playability testing:
I think most of the value will come from the hardening category.
1
0
Excellent clarification - I agree completely. Do you think it would also be in our remit to publish encoding guidelines for mobile optimised content?
0
0
I like this idea, but how would the poor user know why a file doesn't work? We either need some kind of tool the user runs to extract metadata about the file, or an upload servie where the actual problematic file is uploaded. The latter is obviously a copyright nightmare.
0
0
I didn't quite understand your comment - but I think maybe you are assuming this database is targeting end-users? If so, that was not the intention - it is a service for OEMs, developers etc to pool their resources during device testing. The openness of the Symbian community means we are much better positioned than rival OSes to coordinate the distribution of this work, and coordination is the key to reduce costs and increase test coverage.
0
0
I love the idea. Providing test suites is always a good idea. For multimedia, this can be difficult and lends itself very well to a collaborative contributions-based approach. I think I see Sebastian's point about "not being able to be sure if it worked". If a file plays, but plays incorrectly, it can be hard to tell there was a failure - simple example: if a file for some audio format plays incorrectly in mono rather than stereo, you won't know it's an error unless there's some kind of note (or more formal metadata linked to a test-running framework) that says it should be stereo. But, even with this deficiency in the testing, at least you will catch the (more common) cases where the file simply doesn't play, or where the failure is otherwise obvious. But this "how can you tell if it worked" problem exists for all sorts of multimedia testing (and a lot of other user-experience testing) ... in a former life (Chief Test Architect for symbian.com) I've even asked the same question of people selling well-known expensive commercial format-compatibility test suites and found they didn't have very satisfactory answers ("Well, you just view all the different files and, uh, see if there are any problems.") So, putting this shared test-technique problem aside, the core idea of having a colllaboratively sourced set of multimedia test files is wonderful, and simple. And, as Brendan suggests, it can be nicely extended to test file-formats outside the multimedia area.
0
0
Ah I think I see the confusion now. The key issue about use-case testing is that it most tests can't be automated - unless the test causes a crash, it requires human eyes and ears. For example, an issue where AV sync drifts is a human perception - the platform won't be able to report the drift, since as far as it is concerned audio and video are still in sync! To get to the holy grail of multimedia "just working" requires much effort of this variety. Hence although this idea may not fit with everyone's immediate idea of Symbian testing, the Symbian community can help share this cost and hence build a competitive advantage. Does that make my idea clearer?
0
0
> Does that make my idea clearer? I think so. Not trying to solve the larger "How can we tell (using automated techniques if possible) if it's playing correctly?" problem, but a collaborative repository for the simpler "How do we get enough examples of different formats for testing playability?" problem. A solid, practical, good idea.
0
0
Good idea, though I suspect that device manufacturers already have sets of media files (including "horror" ones) that they use in their internal testing. For example, in the old Symbian Software Ltd. we had lots of JPEG/EXIF images that we tested including ones with deliberate, nasty corruptions that could crash your phone if the decoder wasn't well implemented. I'd be very surprised if people didn't do the same for audio and video files. What I think might be useful is to define some detailed, minimum requirements (codecs, profiles, max bitrates etc. etc.) for media support on Symbian devices along with a suite of test files that should play when those requirements are fulfilled. I think that would make life much easier for content distributors since rather than having to figure out what works and what doesn't on each device they could rely on the fact that, as long as their content is encoded according these parameters, it is guaranteed to play perfectly on any Symbian device from any manufacturer. Vendors of audio/video transcoding software would benefit too since they could add a "Symbian phone" preset rather than having to have lots for the various devices (or just not bothering at all). The minimum media support requirements could be revised from time to time and perhaps linked to Symbian releases. E.g. every Symbian^4 device can play fooCodec files at nHD@30fps. Every Symbian^5 device can do everything the Symbian^4 ones could plus it now supports barCodec too. Getting all the device manufacturers to agree on a minimal set would be tough no doubt, but I recon it might be worth a try. James Nash: arty phone geek
0
0
First comment (not aimed at James at all!) - this concept of "nHD" is, erm, interesting. I wish I could mandate a huge health warning be put on it - since "nHD" is actually a lower resolution than standard definition (D1)! This comment is definitely intended light heartedly, but I think the mobile industry might paint itself into a corner with this one... But back to business - I agree with the concept of a reference set of MM use-case tests as a means of providing some assurance of playability robustness. I do however think it might be an unattaiable goal, since there will always be something that doesn't play and that would undermine the assurances the reference suite was trying to provide. The other issue is that there will be issues that aren't covered by such a reference spec - for example, the Microsoft Encoder for ASF files assumes infinite video ES buffer. This can cause problems with devices that have limited the size of this buffer for whatever reason. Nothing in the reference spec would guard against this issue.
0
0
I recently talked to someone from a localization test house in Finland. They could benefit as well from this as it is an area of testing that is difficult to automate (according to them). It might be good Martin if, before proceeding with the implementation of this idea, we can collect together a list of areas where this approach would suit. What do you think?
0
0
I don't mind that, within reason - since I would want to avoid analysis paralysis from trying to make a gold-plated system. I'm pretty sure the infrastructure behid this is actually the simplest part of this. I'm currently working with Dennis to scope it and produce a rough costing. The biggest effort will be in selling the idea to the community and getting them to use it. As a result, I don't think much will be lost by implementing something that needs slight modifications to work in a different context. Interestingly, two other contexts have emerged that would benefit from this kind of approach: - UI testing - "plugfest" testing - e.g. trying with as many bluetooth headsets as possible
0
0
Yes, I had thought of UI testing and device interoperability is another great one (could go beyond Bluetooth to say USB devices). And with the greatest of respect to your excellent idea, these areas also don't suffer from some of the possible IP issues of the original idea as they would involve just a text description of the use-case.
0
0
This has some good points. Please review, comment and vote, if applicable.
0
0
Bringing back for community support!
0
0
You'll all be pleased to know the database infrastructure has been contributed by a company called Comarch, and we are in the process of integrating it!
The database will be used by a number of community testing projects - more information will soon be published on the Symbian blog, so stay tuned.
0
0
|
Sign in or Register
Idea Stats
Idea Approvals
Team Members
Similar Ideas
Who Else is Viewing0 Members, 0 guests
Email Group |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
/ 0 