Friday, 11 October 2013

Want better <-> Amarok sync? Let's tell them!

Heya folk,
I need your help. Amarok has been able to sync personal metadata like play count, rating, labels with your library on since version 2.7. Unfortunately one of the most valuable piece of information, last played time (and also first played time) still cannot be synced due to minor omission in Web Service API.

Yesterday I've seen third user requesting this feature. That was enough for me. Let them hear our needs! Please vote on the API suggestion I've submitted to provide last played time. No registration needed, just pick your nick. You can also vote on an existing suggestion to also provide first played time.

If there is enough of us, the chances are high that will implement these suggestions and I'll be able to make statistics synchronization with even more useful in Amarok 2.9.

Tuesday, 24 September 2013

Amarok MTP (Android) GSoC: Final Report

Ahoy, it is the time of the year again. The Google Summer of Code has just ended and here I come with the final report, joining my fellow Amarok GSoC student Konrad Zemek with his excellent posting.

Newer MTP device configuration dialog
Before I sum things up, let me quickly mention what I've done the very last week. In short, it was full of little features and polishing.

What I've done this week:
  • Fixed small glitches in playlist tooltips (properly escape HTML markup) and in MemoryMeta (show non-compilation albums with empty album artist).
  • Implemented reading of the device folder structure, showing the actual track path as its location (the format is compatible with kio-mtp):
  • Implemented better fall-back when we fail to read device's storage entries (mostly because the Android device has its screen locker). It turns out that (at least Samsung's MTP stack) only blocks storage listing, but happily lists all the files and other entries. So we exploit that and infer storage ids from them. :-)
  • Added support for specifying the Music folder, the folder where new music is put into. I was trying a handful of approaches how to make this editable by user and ended up with an editable combo box that shows root items and provider nice completion for all folders. I had to implement full QAbstractItemModel of MTP folder because of this, but it was fun. ;-)
  • Finally added option to show only tracks under the Music folder. This is handy in order not to pick up misc music files like ringtones.
Problems I've faced:
  • KCompletion is weird and subclassing it to my liking seems impossible because few methods are virtual and other ones use private d-pointer extensively. Fortunately QCompleter is much nicer and works (pretty well) even with a hierarchic model.
  • MTP folder/storage model is weird. There are storages (e.g. Internal memory vs. SD card) and folders, so in order to remember "put things here" you have to save either storage or folder id and whether it is a storage or a folder. I guess this is because it was designed by Microsoft...

Wrapping up

Let's see if the goals outlined in the initial project proposal have been met:
  • Complete MTP Collection rewrite that removes the dependency on deprecated MediaDevice framework: Check
  • Actual communication with the device is strictly asynchronous (threaded) in order not to block the UI: Check
  • Zero-configuration device detection and enumeration, plug & play: Check
  • Enumeration of tracks on the device along with their metadata: Check
  • Playback of tracks stored on devices with on-demand background loading: Check
  • Ability to transfer tracks out of and to MTP devices, with possibility to transcode: Check
  • Full playlist management (for devices that support them): No, see below

Future work

  • The problem with playlists is that the ones created in the most popular music player on Androids, the Google Play Music, aren't visible through the MTP API. It needs further investigation whether and how these could be accessed. In short-term I'll try to add support for traditional playlists, but I'll need external testers for that.
  • I haven't touched the album API at all, because my device (S III Mini with Android 4.2) happily ignores it and shows albums correctly without it. It might be however needed for proper album support on older devices. If it turns out this is the case during user testing, I'll add support for it.
  • Ability to specify custom file naming when uploading music would be nice.
  • Album art. My device displays album art embedded in file tags and I plan to add generic support to embed these to our transcoding/copying framework, but we could make use of the MTP thumbnail API.
My current plan is to add the last round of features and to do some final polishing, then submit a review request. As soon as the work is merged into master I'll request testing here. You can of course (and are encouraged to) test my work right away by checking out gsoc branch of my personal Amarok clone repository.

Sunday, 15 September 2013

Amarok MTP (Android) GSoC week 13: Transcoding

Hey, this is my 13th weekly report describing my work on my Google Summer of Code project to rewrite MTP (Android) support in Amarok from scratch. The big item is week is transcoding.

On-the-fly transcoding has come to Amarok's Android (MTP) collection.
What I've done this week:
  • Implemented a little configuration dialog that shows some information and lets you set some preferences (more in the future). Accessible from Collection Browser:
    The little wrench opens the configuration dialog.
  • Implemented support for renaming the device (setting friendly name). The name is actually sent to the device and is also picked up by kio-mtp etc.
    MTP device configuration dialog. Notice the Transcode label.
  • Implemented on-the-fly transcoding when copying/moving tracks to the device, yay! Your transcoding preferences can optionally be remembered (as is usual in Amarok) and the encoding even makes use of more than one of your processor cores. See:
This is the proof that transcoding works. :-)
  • Done small tweaks like correctly releasing the device when Amarok is closed, trying not to confuse user, improving feedback...
Problems I've faced:
  • A known bug where transcoding a track with embedded cover image creates a video ogg file, the metadata thereof (most importantly file size) are then refused to be read by Amarok.
  • Infinite loop in libmtp when the actual size of the file you want to upload is different than what you claim. Complements interestingly the above bug.
What's next:
  • Ability to see & specify file structure of the on-device tracks.
  • Polishing.
You can view and test my code by checking out gsoc branch of my personal Amarok clone repository.

Wednesday, 11 September 2013

Amarok MTP (Android) GSoC: weeks 11 & 12 - Full Sync!

Yay, I've achieved a significant goal in my GSoC project to rewrite MTP (Android) support in Amarok from scratch. Yes, it is the ability to fully manage tracks on your MTP devices.
Amarok copying tracks to Android device
What I've done last weeks:
  • Added fancy progress bars when downloading/uploading/updating tracks on the MTP device. The progress bars are "cumulative", which means that there is only one progress bar for a given type of operation and device. Potential new jobs are added to it when it is already running.
  • Reworked locking in order not to hold 2 locks simultaneously. There are N + 2 locks in each MTP collection (where N is the number of tracks) and holding any 2 of them simultaneously creates a potential deadlock situation (unless lock order is preserved) and may stall the UI for significant time. I managed to sneak in a fancy static (compile-time) assertion because I had to provide a copy constructor for a libmtp library object.
  • Implemented removal of tracks on MTP device. This was rather easy, also comes with a progress feedback, but it is too fast to be screeshot-able. :)
  • And finally implemented copying/moving tracks to the MTP device. This means that Amarok is now able to fully manage audio content of your Android phones and other MTP devices! It took a bit of effort, but it works nicely now. I've even tested crazy things like plugging the device out in the middle of a move operation to check the behaviour is correct (it is and Amarok doesn't crash nor eat your kittens (YMMV)).
What's next:
  • Allowing to specify a folder structure and show human-readable folders for existing tracks.
  • A config dialog for each device.
  • Transcoding (low-level part is virtually done, only needs UI).
You can view and test my code by checking out gsoc branch of my personal Amarok clone repository.

Tuesday, 3 September 2013

Amarok MTP (Android) GSoC: week 10

Tracks being copied from Android to iPod
Hi, this is my (a bit late *cough*) report from week 10 describing my work on my Google Summer of Code project to rewrite MTP (Android) support in Amarok from scratch. The report from week 11 will follow shortly, but I'll keep them split. In week 10 I've mainly worked on copy-ability of tracks out of the MTP device.

What I've done this week:
  • Reworked how MTP devices are initialized. The device is no longer shown in the Collection Browser with 0 tracks while it is being initialized, an (abort-able) progress bar is shown instead. This is also a pre-requisite of the point below...
  • Implemented restoring of MTP tracks from their uid. This means that if you quit Amarok with some MTP tracks in you playlist, they turn from grayed-out to playable once you plug in your device on next Amarok run. It also means that Saved Playlists (in Amarok db) with MTP tracks now work.
  • Implemented copying of tracks from the MTP device to other Amarok collections (i.e. the first half of MtpCollectionLocation), yay! It works quite well here, although I need to add some progress bars.
What's next:
  • Copying tracks to the device, also removing tracks from the device (easy).
  • Config dialog.
You can view and test my code by checking out gsoc branch of my personal Amarok clone repository.

Friday, 16 August 2013

Amarok 2.8 Released! (+MTP (Android) GSoC: weeks 8 & 9)

the day has come. Amarok 2.8 is out! I know I've already said it about 2.7, but this is the best Amarok release. Ever. The number of improvements and bugfixes is insane. The official release announcement contains the most visible changes and the ChangeLog list them all, but let me highlight some that didn't fit into the announcement. We how (also) have:
  • Opus codec support including transcoding by Martin Brodbeck (only with development version of TagLib or 1.9 once it is released though).
  • ASX playlist support by Tatjana Gornak (used by many on-line radios).
  • Improved Nepomuk Collection that dynamically queries Nepomuk by Edward Toroshchin (still in beta state though).
  • Fixed Local Collection and Collection Browser that reflects removed files and folders immediately.
  • More keyboard shortcuts and possible UI workflows.
  • Improved and fixed plugin.
  • More control over transcoding.
  • Improved File Browser.
We've changed a couple of existing things too:
  • We've removed the splash screen, with an SSD is it just an unnecessary flicker.
  • Album Artist level have replaced Artist one in the Collection Browser by default; the Artist level no-longer duplicates tracks under Various Artists.
  • Playback start on double-clicking has been made configurable, middle-click action to play tracks immediately has been added.
Apart from releasing Amarok, I've also done some work on my GSoC project, although the amount of it has been limited by my other responsibilities. Here comes the report for my project to rewrite MTP (Android) support in Amarok from scratch. I've mainly perfected editing of track metadata.

What I've done last weeks:
  • Finished support for editing track metadata in a way that I'm satisfied with. It is a bit elaborate, but anything less would be suboptimal, see:
    1. User changes some metadata.
      • we note the set of the metadata that she actually changed.
    2. Background job to process the change is initiated.
    3. MTP commands to update the properties are sent to the device. If the user has disabled Use File Metadata, the job is finished. Else...
    4. If the track is not yet cached locally, it is downloaded to the local filesystem.
      • Because the file tags are usually more accurate than the metadata the MTP device provides us, we update the current metadata with file tags, except the ones that were changed.
    5. Tags in the local cached file are updated using TagLib.
    6. The local cached file is (re)uploaded to the device under a temporary name with a random part in it.
    7. The old on-device track object is deleted
    8. The temporary on-device file is renamed back to the original one.
Problems I've faced:
  • There doesn't seem to be a way with MTP to replace contents of an existing object. This means that the item id of a existing tracks can change, which is unpleasant, but manageable.
What's next:
  • Config dialog, and finally transferring tracks to the device! (it is in fact half-done, because the code will be shared with track editing support)
You can view and test my code by checking out gsoc branch of my personal Amarok clone repository.

Monday, 5 August 2013

Amarok MTP (Android) GSoC: weeks 6 & 7

I was so busy coding in the last 2 weeks that I forgot a write a report. Here comes a combined report for weeks 6 and 7 concerning work on my Google Summer of Code project to rewrite MTP (Android) support in Amarok from scratch.
What I've done this week:
  • Changed MTP track cache path to be the "cache" directory (usually "/var/tmp/kdecache-$username/") instead of the "tmp" directory (usually /tmp/kde-$username/). The cache can get very big (gigabytes) in extreme cases and /tmp might be a size-constrained memory-backed filesystem.
  • Added code to explicitly remove stale files in MTP track cache. The relevant files are normally removed when the associated track is not referenced any more (i.e. when the device in unplugged), but for example when a MTP track remains in the playlist, it may be still referenced at the time Amarok quits.
  • Introduced support for keeping per-device MTP collection configuration (not yet represented in the UI).
  • Added first configuration option and associated feature: reading and writing file tags (in addition to MTP metadata; useful because at least my Samsung S II mini doesn't seem to be able to extract year and track number out of some files). This means that when a track file is available, tags are read from it, enhancing existing (MTP) tags. In future, it will work conversely when metadata are edited.
  • The trick from the previous post worked - track playback is started as soon as ~1 MB of it is cached. This means that playing even hours-long pieces straight off the devices should be instant.
  • Finally implemented edit-ability of MTP track metadata. It means that you can edit track tags in Amarok and it will tell the device to change them. See also What's next.
What's next:
  • According to my testing, at least my S III mini is able to remember track metadata we send to it, but it doesn't write it down to the actual files. I'll implement optional support to download, change and upload the file in metadata change.
  • Implementing a small dialog with configuration options and some device & troubleshooting info.
You can view and test my code by checking out gsoc branch of my personal Amarok clone repository.