Friday, 11 October 2013

Want better <-> Amarok sync? Let's tell them!

Heya folk,
I need your help. Amarok has been able to sync personal metadata like play count, rating, labels with your library on since version 2.7. Unfortunately one of the most valuable piece of information, last played time (and also first played time) still cannot be synced due to minor omission in Web Service API.

Yesterday I've seen third user requesting this feature. That was enough for me. Let them hear our needs! Please vote on the API suggestion I've submitted to provide last played time. No registration needed, just pick your nick. You can also vote on an existing suggestion to also provide first played time.

If there is enough of us, the chances are high that will implement these suggestions and I'll be able to make statistics synchronization with even more useful in Amarok 2.9.

Tuesday, 24 September 2013

Amarok MTP (Android) GSoC: Final Report

Ahoy, it is the time of the year again. The Google Summer of Code has just ended and here I come with the final report, joining my fellow Amarok GSoC student Konrad Zemek with his excellent posting.

Newer MTP device configuration dialog
Before I sum things up, let me quickly mention what I've done the very last week. In short, it was full of little features and polishing.

What I've done this week:
  • Fixed small glitches in playlist tooltips (properly escape HTML markup) and in MemoryMeta (show non-compilation albums with empty album artist).
  • Implemented reading of the device folder structure, showing the actual track path as its location (the format is compatible with kio-mtp):
  • Implemented better fall-back when we fail to read device's storage entries (mostly because the Android device has its screen locker). It turns out that (at least Samsung's MTP stack) only blocks storage listing, but happily lists all the files and other entries. So we exploit that and infer storage ids from them. :-)
  • Added support for specifying the Music folder, the folder where new music is put into. I was trying a handful of approaches how to make this editable by user and ended up with an editable combo box that shows root items and provider nice completion for all folders. I had to implement full QAbstractItemModel of MTP folder because of this, but it was fun. ;-)
  • Finally added option to show only tracks under the Music folder. This is handy in order not to pick up misc music files like ringtones.
Problems I've faced:
  • KCompletion is weird and subclassing it to my liking seems impossible because few methods are virtual and other ones use private d-pointer extensively. Fortunately QCompleter is much nicer and works (pretty well) even with a hierarchic model.
  • MTP folder/storage model is weird. There are storages (e.g. Internal memory vs. SD card) and folders, so in order to remember "put things here" you have to save either storage or folder id and whether it is a storage or a folder. I guess this is because it was designed by Microsoft...

Wrapping up

Let's see if the goals outlined in the initial project proposal have been met:
  • Complete MTP Collection rewrite that removes the dependency on deprecated MediaDevice framework: Check
  • Actual communication with the device is strictly asynchronous (threaded) in order not to block the UI: Check
  • Zero-configuration device detection and enumeration, plug & play: Check
  • Enumeration of tracks on the device along with their metadata: Check
  • Playback of tracks stored on devices with on-demand background loading: Check
  • Ability to transfer tracks out of and to MTP devices, with possibility to transcode: Check
  • Full playlist management (for devices that support them): No, see below

Future work

  • The problem with playlists is that the ones created in the most popular music player on Androids, the Google Play Music, aren't visible through the MTP API. It needs further investigation whether and how these could be accessed. In short-term I'll try to add support for traditional playlists, but I'll need external testers for that.
  • I haven't touched the album API at all, because my device (S III Mini with Android 4.2) happily ignores it and shows albums correctly without it. It might be however needed for proper album support on older devices. If it turns out this is the case during user testing, I'll add support for it.
  • Ability to specify custom file naming when uploading music would be nice.
  • Album art. My device displays album art embedded in file tags and I plan to add generic support to embed these to our transcoding/copying framework, but we could make use of the MTP thumbnail API.
My current plan is to add the last round of features and to do some final polishing, then submit a review request. As soon as the work is merged into master I'll request testing here. You can of course (and are encouraged to) test my work right away by checking out gsoc branch of my personal Amarok clone repository.

Sunday, 15 September 2013

Amarok MTP (Android) GSoC week 13: Transcoding

Hey, this is my 13th weekly report describing my work on my Google Summer of Code project to rewrite MTP (Android) support in Amarok from scratch. The big item is week is transcoding.

On-the-fly transcoding has come to Amarok's Android (MTP) collection.
What I've done this week:
  • Implemented a little configuration dialog that shows some information and lets you set some preferences (more in the future). Accessible from Collection Browser:
    The little wrench opens the configuration dialog.
  • Implemented support for renaming the device (setting friendly name). The name is actually sent to the device and is also picked up by kio-mtp etc.
    MTP device configuration dialog. Notice the Transcode label.
  • Implemented on-the-fly transcoding when copying/moving tracks to the device, yay! Your transcoding preferences can optionally be remembered (as is usual in Amarok) and the encoding even makes use of more than one of your processor cores. See:
This is the proof that transcoding works. :-)
  • Done small tweaks like correctly releasing the device when Amarok is closed, trying not to confuse user, improving feedback...
Problems I've faced:
  • A known bug where transcoding a track with embedded cover image creates a video ogg file, the metadata thereof (most importantly file size) are then refused to be read by Amarok.
  • Infinite loop in libmtp when the actual size of the file you want to upload is different than what you claim. Complements interestingly the above bug.
What's next:
  • Ability to see & specify file structure of the on-device tracks.
  • Polishing.
You can view and test my code by checking out gsoc branch of my personal Amarok clone repository.

Wednesday, 11 September 2013

Amarok MTP (Android) GSoC: weeks 11 & 12 - Full Sync!

Yay, I've achieved a significant goal in my GSoC project to rewrite MTP (Android) support in Amarok from scratch. Yes, it is the ability to fully manage tracks on your MTP devices.
Amarok copying tracks to Android device
What I've done last weeks:
  • Added fancy progress bars when downloading/uploading/updating tracks on the MTP device. The progress bars are "cumulative", which means that there is only one progress bar for a given type of operation and device. Potential new jobs are added to it when it is already running.
  • Reworked locking in order not to hold 2 locks simultaneously. There are N + 2 locks in each MTP collection (where N is the number of tracks) and holding any 2 of them simultaneously creates a potential deadlock situation (unless lock order is preserved) and may stall the UI for significant time. I managed to sneak in a fancy static (compile-time) assertion because I had to provide a copy constructor for a libmtp library object.
  • Implemented removal of tracks on MTP device. This was rather easy, also comes with a progress feedback, but it is too fast to be screeshot-able. :)
  • And finally implemented copying/moving tracks to the MTP device. This means that Amarok is now able to fully manage audio content of your Android phones and other MTP devices! It took a bit of effort, but it works nicely now. I've even tested crazy things like plugging the device out in the middle of a move operation to check the behaviour is correct (it is and Amarok doesn't crash nor eat your kittens (YMMV)).
What's next:
  • Allowing to specify a folder structure and show human-readable folders for existing tracks.
  • A config dialog for each device.
  • Transcoding (low-level part is virtually done, only needs UI).
You can view and test my code by checking out gsoc branch of my personal Amarok clone repository.

Tuesday, 3 September 2013

Amarok MTP (Android) GSoC: week 10

Tracks being copied from Android to iPod
Hi, this is my (a bit late *cough*) report from week 10 describing my work on my Google Summer of Code project to rewrite MTP (Android) support in Amarok from scratch. The report from week 11 will follow shortly, but I'll keep them split. In week 10 I've mainly worked on copy-ability of tracks out of the MTP device.

What I've done this week:
  • Reworked how MTP devices are initialized. The device is no longer shown in the Collection Browser with 0 tracks while it is being initialized, an (abort-able) progress bar is shown instead. This is also a pre-requisite of the point below...
  • Implemented restoring of MTP tracks from their uid. This means that if you quit Amarok with some MTP tracks in you playlist, they turn from grayed-out to playable once you plug in your device on next Amarok run. It also means that Saved Playlists (in Amarok db) with MTP tracks now work.
  • Implemented copying of tracks from the MTP device to other Amarok collections (i.e. the first half of MtpCollectionLocation), yay! It works quite well here, although I need to add some progress bars.
What's next:
  • Copying tracks to the device, also removing tracks from the device (easy).
  • Config dialog.
You can view and test my code by checking out gsoc branch of my personal Amarok clone repository.

Friday, 16 August 2013

Amarok 2.8 Released! (+MTP (Android) GSoC: weeks 8 & 9)

the day has come. Amarok 2.8 is out! I know I've already said it about 2.7, but this is the best Amarok release. Ever. The number of improvements and bugfixes is insane. The official release announcement contains the most visible changes and the ChangeLog list them all, but let me highlight some that didn't fit into the announcement. We how (also) have:
  • Opus codec support including transcoding by Martin Brodbeck (only with development version of TagLib or 1.9 once it is released though).
  • ASX playlist support by Tatjana Gornak (used by many on-line radios).
  • Improved Nepomuk Collection that dynamically queries Nepomuk by Edward Toroshchin (still in beta state though).
  • Fixed Local Collection and Collection Browser that reflects removed files and folders immediately.
  • More keyboard shortcuts and possible UI workflows.
  • Improved and fixed plugin.
  • More control over transcoding.
  • Improved File Browser.
We've changed a couple of existing things too:
  • We've removed the splash screen, with an SSD is it just an unnecessary flicker.
  • Album Artist level have replaced Artist one in the Collection Browser by default; the Artist level no-longer duplicates tracks under Various Artists.
  • Playback start on double-clicking has been made configurable, middle-click action to play tracks immediately has been added.
Apart from releasing Amarok, I've also done some work on my GSoC project, although the amount of it has been limited by my other responsibilities. Here comes the report for my project to rewrite MTP (Android) support in Amarok from scratch. I've mainly perfected editing of track metadata.

What I've done last weeks:
  • Finished support for editing track metadata in a way that I'm satisfied with. It is a bit elaborate, but anything less would be suboptimal, see:
    1. User changes some metadata.
      • we note the set of the metadata that she actually changed.
    2. Background job to process the change is initiated.
    3. MTP commands to update the properties are sent to the device. If the user has disabled Use File Metadata, the job is finished. Else...
    4. If the track is not yet cached locally, it is downloaded to the local filesystem.
      • Because the file tags are usually more accurate than the metadata the MTP device provides us, we update the current metadata with file tags, except the ones that were changed.
    5. Tags in the local cached file are updated using TagLib.
    6. The local cached file is (re)uploaded to the device under a temporary name with a random part in it.
    7. The old on-device track object is deleted
    8. The temporary on-device file is renamed back to the original one.
Problems I've faced:
  • There doesn't seem to be a way with MTP to replace contents of an existing object. This means that the item id of a existing tracks can change, which is unpleasant, but manageable.
What's next:
  • Config dialog, and finally transferring tracks to the device! (it is in fact half-done, because the code will be shared with track editing support)
You can view and test my code by checking out gsoc branch of my personal Amarok clone repository.

Monday, 5 August 2013

Amarok MTP (Android) GSoC: weeks 6 & 7

I was so busy coding in the last 2 weeks that I forgot a write a report. Here comes a combined report for weeks 6 and 7 concerning work on my Google Summer of Code project to rewrite MTP (Android) support in Amarok from scratch.
What I've done this week:
  • Changed MTP track cache path to be the "cache" directory (usually "/var/tmp/kdecache-$username/") instead of the "tmp" directory (usually /tmp/kde-$username/). The cache can get very big (gigabytes) in extreme cases and /tmp might be a size-constrained memory-backed filesystem.
  • Added code to explicitly remove stale files in MTP track cache. The relevant files are normally removed when the associated track is not referenced any more (i.e. when the device in unplugged), but for example when a MTP track remains in the playlist, it may be still referenced at the time Amarok quits.
  • Introduced support for keeping per-device MTP collection configuration (not yet represented in the UI).
  • Added first configuration option and associated feature: reading and writing file tags (in addition to MTP metadata; useful because at least my Samsung S II mini doesn't seem to be able to extract year and track number out of some files). This means that when a track file is available, tags are read from it, enhancing existing (MTP) tags. In future, it will work conversely when metadata are edited.
  • The trick from the previous post worked - track playback is started as soon as ~1 MB of it is cached. This means that playing even hours-long pieces straight off the devices should be instant.
  • Finally implemented edit-ability of MTP track metadata. It means that you can edit track tags in Amarok and it will tell the device to change them. See also What's next.
What's next:
  • According to my testing, at least my S III mini is able to remember track metadata we send to it, but it doesn't write it down to the actual files. I'll implement optional support to download, change and upload the file in metadata change.
  • Implementing a small dialog with configuration options and some device & troubleshooting info.
You can view and test my code by checking out gsoc branch of my personal Amarok clone repository.

Tuesday, 23 July 2013

Amarok MTP (Android) GSoC: week 5

Ahoj, this is my fifth weekly report about my work on a Google Summer of Code project to rewrite MTP (Android) support in Amarok from scratch. Last week was also affected by me attending Akademy, so I'll also talk about some of my Amarok-related work there. Fortunately I've still managed to fulfil my GSoC goal scheduled for week 5: playability.
Amarok playing Daft Punk's piece straight off my S III Mini
What I've done last week:
  • Fixed long-standing and severe track organizing bugs on Windows in cooperation with Patrick von Reth. Thanks, Patrick!
  • Sat down with Àlex Fiestas and debugged a solid bug that prevents Amarok from seeing your USB storage drives and iPods that are connected at the time Amarok starts.
  • Fixed a couple of bugs that I've introduced shortly before releasing 2.8 Beta, one of which was unfortunately severe.
  • Finally implemented playability of MTP tracks in my GSoC branch. Unfortunately not as asynchronously as I wanted, because phonon (at least phonon-gstreamer 4.6.3) cannot cope with tracks that are still being downloaded, so I do some "asynchronous with blocking waiting in the main thread with a timeout" instead.
  • Hmm, actually now I get an idea how we could fool Phonon. We know file size in advance, so perhaps we can pre-size the temporary file at the beginning of the transfer (which will also help fragmentation). Needs testing tough.
Problems I've faced:
  • As said above, Phonon needing the entire file before starting playback (or perhaps there's another trick?).
  • Amarok's Meta::Track API is not really being suited for tracks that want to asynchronously load themselves before playing.
What's next:
  • Editing track metadata and more overall polish.
You can view and test my code by checking out gsoc branch of my personal Amarok clone repository.

Wednesday, 17 July 2013

Amarok MTP (Android) GSoC: week 4; hello from Bilbao!

Kaixo, this is my fourth weekly report covering my Google Summer of Code project to rewrite MTP (Android) support in Amarok from scratch. Code-wise, I have done only a couple of small commits the last week, but this is fine because I've already done the work scheduled for week 4 during week 3. :-)
Photo from the GSoC, GCI, SOK & OPfW students & mentors BoF (which actually well exceeded my expectations). Can you spot me? :-)
I think you've already guessed why I've done a bit less than usual and published the weekly report late: I'm enjoying the great atmosphere of Akademy 2013 here in Bilbao, Spain.

The event rocks so hard (with all the talks, BoFs, chats, parties, concerts, trips and hacking going on) that it is impossible to describe in a blog post. More importantly for my GSoC project, I've met up with Philipp Schmidt, the kio-mtp author (Àlex Fiestas wants to tell you he did just the screencast), who knows all the quirks of using MTP devices in KDE and is happy to share them. We've also sat down and tweaked the proposed MTP Daemon D-Bus interface, with even more daring suggestions from me being in preparation.
Sunset over ocean near Bilbao taken during coast walk. Thanks go to Dani for organizing it!
What I've done last week:
  • Improved Solid <-> libmtp device matching by shamelessly copying a bit of Philipp's kio-mtp code. (okay, he actually pro-actively showed me the code, so it was not that shameless)
  • Removed extra device check that actually prevented Amarok from detecting a Sansa Clip device that Matija generously lent me to test it. Thanks, Matija! [note to self: need to discuss this with Phillip]
  • Had a serious look at the proposed MTP Daemon interface together with Philipp, sent out first 2 patches for it for review.
Problems I've faced:
  • Partying too hard, getting to bed past 4 AM, oversleeping and missing the morning BoFs.
What's next:
  • Implementing playability of the tracks. Should be easy with all the MTP knowledge right next to me in form of Philipp.
You can view and test my code by checking out gsoc branch of my personal Amarok clone repository.

Let's repeat it,
I'm here till Friday, ping strohel on #amarok or #akademy if you want to talk about anything Amarok-related with me in person.

Sunday, 7 July 2013

Amarok MTP (Android) GSoC: week 3; Amarok 2.8 Released

Hola, this is a third weekly report describing my work on my Google Summer of Code project to rewrite MTP (Android) support in Amarok from scratch. This week I've finished implementing reading the track list from MTP devices.
What I've done this week:
  • Finished the background job to read MTP tracks and their metadata. Includes a nice progress bar (everyone loves progress bars) and is abort-able too.
  • Implemented central MtpTrack class and associated MtpAlbum, MtpArtist, MtpComposer, MtpGenre and MtpYear classes that are used to represent the collection entities in the Collection Browser.
  • Thanks to the above, Collection Browser now shows the entity tree of your MTP device (first screen-shot), which is nice.
Problems I've faced:
  • First little bugs/API omissions with libmtp are showing up (inability to flush libmtp's object cache). Fortunately the upstream is very helpful, so I'm looking forward to discuss it with them.
What's next:
  • Given that I've basically done work of both week 3 & week 4 from my timeline, I'll do a bit of polishing and testing. I'll also look at week 5 schedule which contains playability of MTP tracks directly in Amarok.
You can view and test my code by checking out gsoc branch of my personal Amarok clone repository.

On a related note, we've released Amarok 2.8 Beta! Yay! It contains a galore of improvements and corrections that are slated to end up in Amarok 2.8 (final). I definitely think you can give it some testing, can you? :-)

Sunday, 30 June 2013

Amarok MTP (Android) GSoC: week 2

Heya, this is my second weekly report describing my work on my Google Summer of Code project to rewrite MTP (Android) support in Amarok from scratch. This week I've done a lot of work on MtpCollection class regarding device initialization.
Information for Android users
What I've done this week:
  • MTP devices detected by Solid are actually matched with the ones that libmtp sees.
  • Created an asynchronous job to initialize the MTP device for use by Amarok (query vendor, product, serial number, supported file types, ...).
  • Created an asynchronous job to list device storage drives.
  • Added a refresh button to trigger rebuilding the list of available storage and the list of tracks (in future).
  • Added support for showing device capacity and free space:
    Everyone loves the nice capacity bars
  • Started working on a job to parse tracks found on the device (not presented in the UI yet).
Problems I've faced:
  • Forgetting to call LIBMTP_Initialize() leads to strange errors. :-)
What's next:
  • Diving into full meta system: MtpTrack, album, artist, ...
You can view and test my code by checking out gsoc branch of my personal Amarok clone repository. And finally...
Yay, I'll be there!

Tuesday, 25 June 2013

Amarok MTP GSoC: week 1

Hi, this is my first weekly report describing my work on my Google Summer of Code project to rewrite MTP (Android) support in Amarok from scratch. This week I've laid the very basic building blocks and I even have some screen-shots. :-)
Entries for the newly (re)written MTP collections show up at appropriate places
What I've done this week:
  • Got rid of the old MTP collection implementation and cleaned up some stuff that is no longer needed
  • Started new MTP implementation by coding MtpCollectionFactory, which is responsible for creating collections for devices as they are plugged in and destroying collections for removed devices. This works fine now and the class is 95% done now.
  • Created stub MtpCollection class so that we can show something in the Collection Browser.
  • Created .desktop file for Solid so that an action to open MTP devices with Amarok shows up in Plasma Device Notifier, see below.
    Device Notifier showing Amarok action
Problems I've faced:
  • None yet, they are expected later. :-)
What's next:
  • Great deal of work on MtpCollection regarding track and other metadata parsing, libmtp interaction, threading, locking...
You can view and test my code by checking out gsoc branch of my personal Amarok clone repository.

Tuesday, 28 May 2013

Exciting Summer for Amarok

Since 2005, summers are slightly more interesting for open-source projects than other seasons, and this year will fortunately be no different. You've guessed right, the reason is Google Summer of Code! I'm super-happy about the students that were accepted to hack on Amarok this year. And yes, I'm one of them. :-)

Tatjana Gornak (melandory), who has already refactored the way how we handle playlist files and implemented support for .asx playlists, will rewrite AudioCD support, making it much more solid and solving bugs that have bothered us for ages. She'll also take a look whether it could work on Windows too and remove a dependency on a deprecated framework within Amarok.

Anmol Ahuja (DarthCodus), who has already submitted and got merged 7 patches to Amarok, will revamp Amarok Scripting Interface, improve its documentation, make it tested and will add bindings for more Amarok components so that you will be able to write even more powerful scripts. He has already started surveying script authors for what they would like to see added on Amarok forum, so be sure speak up.

Konrad Zemek (kzemek), finalist of a prestigious Polish programming competition who has already fixed playlist sorting, will improve importing personal metadata from other players. More specifically he'll refactor Amarok 1.4 and iTunes importers so that they match tracks by tags and allow synchronization (rather than overwriting) of metadata. He'll also add support for synchronizing with Rhythmbox and another Amarok 2.x instance.

Yours truly (strohel) will rework MTP collection support from scratch, making sure it works flawlessly, especially on Android phones (not forgetting other MTP devices of course). Apart from fixing long-standing bugs and porting it out from the deprecated framework, new features like on-the-fly transcoding will be added too.

Given that I've had the luck to review past work of all mentioned students, I'm absolutely confident that all the projects will result in a big success.™ Look ahead to Amarok 2.9! (oh, is that speaking about N+2 while N+1 is not yet out syndrome?) :)

There have been 4 more student proposals (some of them very good) for Amarok that we unfortunately couldn't accept, due to lack of mentoring manpower. You're encouraged to keep submitting patches, an excellent preparation for the next time.

Thursday, 16 May 2013

Amarok 2.7.1 Released!

Hi there, while we've been working very hard on the next Amarok feature release, the 2.8, we also haven't forgot the majority of our users using the stable versions.

Welcome Amarok 2.7.1, a very close relative of 2.7.0 with just a couple of very important bug fixes. The 2.7.1 is also an opportunity for Arch Linux to package it correctly. :-)

Don't fear that we've made just 8 commits since 2.7.0. In fact, we've made over 300 by 25 different people! Look forward for better transcoding, ASX playlist and Opus audio format support, improved MusicBrainz tag guessing and improvements of existing features all over the place for Amarok 2.8.0.

Friday, 18 January 2013

Amarok 2.7 Finally Here!

In case you don't know it yet, we've just released Amarok 2.7! This release is called A Minor Tune, partially because the list of new features isn't pages-long (but you still get statistics synchronization and preliminary Nepomuk integration). But I believe it will be remembered for its level of polish and stability - many components like audio CD playback and file browser have been made much more solid, and the number of user-reported problems that were closed since 2.6 approaches stunning 500.

On the other hand, due to the lack of manpower there are still components that deserve much more love than they currently get. One such example is the MTP player plug-in, but don't fear, we (or I) have plans with it. Contributions in any form (patches being best) are of course more than welcome.