by Anastasis Grammenos (gramanas)
This is the final proposal with witch I got accepted as a student in GSoC 2017.
It's been a real pleasure working on this c++ software this summer, I got hands on with a huuuge project and I somewhat understood how things would work in a production environment. I learned a lot about the database and how to interact with it though code, and maybe most importantly I've learned the value of objects in OOP and how to use them correctly to abstract away details that other classes shouldn't care about.
Before I was accepted I was skimming through the code to see what I would have to face in case I got in, and I was really unsure if I'm gonna make it. It turns out it's a lot easier than it seems. You just have to have a plan, work slowly and carefully and take note of your progress. Org Mode really helped me to plan out my algorithms in plain words and to keep track of what I've done so far in the sometimes big files I had to deal with.
My two mentors also helped a lot, each in their own way, both with our private conversations and with their comments on my PRs, and also by contributing code to other areas of mixxx that affected my project. I cate to appreciate OSS a bit more cause of them, and for that I have to thank them!
Througout the summer I made a revamp to the crates subsystem. I redisigned the class that manages crates, I added support for the nesting features and I created a crate filter to use in the library.
These are the PR's I've made for this project:
The last one changed the filter to it's current state in order to work recursively.
All the work I've done is based on gsoc 2016 Library redesign witch at the moment of writing is not yet ready to ship and is in need of further developement. This means that the nested crates feature will be delayed until developement of the library redesign is finished. Witch brings us to my final point.
The current model used for the tree view of the library doesn't support what I call “dynamic expansion”. Dynamic expasion is the ability to insert new items in the tree or change the current ones (rename/remove) without the need for the whole tree to be reloaded (witch results in the folding of the tree).
It also doesn't support internal drag and drop(Drag one crate to another). For this to work the tree items might need to also adapt to support internal drag and drop.
I think the tree item model should be expanded to support those features. This needs to happen carefully due to the fact that the tree item model is used by other library features that don't need the extra stuff (d&d, dynamic expansion) to work.
The following are the ideas for the project, and the devlog where I was writing what happened each week.
A new filter is added in mixxx's search bar. User can now type
crate: and anything after that
will get matched with the crates the user has created, and display the results.
First of all a new search filter node was created to handle the crate filtering. It is called
CrateFilterNode. Since the crates each song is in
can't be found by the
mixxxlibraryfeature provides I had to create a workaround
in order for
toSql() to work.
cratestorage handles all the crate related database access, so I added a few functions there to generate/run the SQL
required both by
searchqueryparser got modified to recognise the
crate keyword and through the
CrateFilterNode generate the correct
SQL query to get the trackIds of the songs matching the crate the user has entered.
This will be developed on top of Library Redesign. With that in mind I'd like to add some notes on the implementation of the Nested Crates.
TreeItemModelwill be used for the nesting purposes.
For the nested crates to work, we need a viable option to save them in the database.
Basically I'll add a table
| ParentID | ChildID | Depth |
exactly as in the example above.
Also I'll add a table with columns
| CrateID | path | where path is a string that will get generated whenever
the user updates the database by creating/moving/renaming/deleting a crate. This is used to format the tree of the crates.
Since the way the user selects crates will change to reflect the way the rest of the library works, here I'll describe how the filter will work for the nested crates.
Consider the following example:
When the user types some letters after
crate: a query is executed that joins the closure table
with the crates table and returns all the crate id's that match this word and all their children.
Then using these id's it finds the tracks and displays them just like any other filter.
The user can narrow down his search either by typing
(Here I assume that the different filters are joined together by
AND) to get everything under rock/instrumental.
To get all the instrumental tracks of this library the user naturally searches for
A similar procedure to the filter is used when selecting a crate from the tree. Instead of relying to the crate's name this time we have access to it's id so we use it instead to find the children.
While the user organises the library, he/she might want to see only the tracks under crate
but not those in the nested crates underneath. For this a special tree item has been implemented next to the
tree that acts like an toggle switch. When it's on the crates are shown recursively using the method I described above, when it's off it's similar to how it used to work, where you only see only the crate and not it's children.
I think a nice way to create the tree is to use the paths of each crate.
Using the closure table I can get all the ancestors of a crate sorted by depth. Then it's just a matter of iterating through them and appending them to the end of a string.
CrateTreeModel will read the paths table and for each crate if it has a previous
crate in it's path insert it there as a child, else insert it at the top.
You can be sure that every time you add a child the parent is already there, because this
whole think is sorted alphabetically (Just like the example with the paths above - In order for
/Metal/Alternative to be added it must come after
May 15 - June 5
June 6 - June 15
June 16 - June 26
June 27 - July 20
July 20 - July 28
July 29 - August 15
August 15 - August 30
In this section I will mark my progress with a weekly report about the stuff I've been up to with mixxx.
Since I am starting this at June 13 here is a summary of what has been done so far:
the now merged PR at daschuer's repo to fix them.
June 12 - June 19
This week has been pretty slow. I got finals at the moment so I spend some of my time studying.
After the merge of the recent commits to the new library redesign a SearchQueryParser test was failing, so since I want to start working on the new filter for the nested crates I had to fix it.
After some discussions we decided to change the filter functionality
from what it used to be, in order to work better with the library query
helper that generates the queries when the user selects something from the new
library column. Used to be that
artist:“” for example would return all the library
since it's just an empty filter. Now in order for the user to be able to narrow down
to songs with NULL artist
artist:“” returns just that (Unknown artist).
This was already implemented but in a slightly wrong way, so the test failed.
The pull request got merged and now I am ready to start working on the crates themselves.
Next will I'll delve into the
CrateFeature code. From what I understand a tree item model is already
in use for the crates, it's just that there are no levels.
June 19 - June 26
I started with modifying the database and adding the aforementioned tables for the crate hierarchy.
Since everything is at a testing phase for now I have them embedded in the code as crate storage functions that create the tables and insert the data to them.
The closure table works really fine, and I've been able to generate paths for the nested crates as well.
The next step for the database is to implement ways to check that the hierarchy follows the rules. (No same name children etc…)
Now my goal is to use the paths to create the tree. For that I've created a
CrateTreeModel that derives from
TreeItemModel and will handle the
drawing of the tree just like
June 26 - July 3
I got the tree structure working with my dummy database. So I have the DB mockup and the UI mockup. The problem is that this is untestable to everyone else but me.
I'll work on solving that by providing functions to insert child crate and such. This is far from the final product, in witch you can freely move crates to other crates to change their parents etc.
I hope by the end of the following week I'll have a work in progress PR.
July 3 - July 10
I opened a PR with the progress I made here I am wondering if the way I do things leaves a big impact on memory.
The next step is to tie the filter the the crates feature. Selecting a crate will generate the corresponding filter search just like the libraryfeature. For starters I'm gonna have to understand the current way crates are being displayed and then alter it.
I am very happy with the progress so far. I hope I'll have the filter working by next week.
July 10 - July 17
Well I haven't touch the filter yet!
There is some discussion regarding
TrackCollection class and how it will interface
For now I've split the
CrateStorage file taking away all the helper classes and
puting them inside
CrateStorageHelper.h and also merged
CrateStorage handles all the data writing and reading regarding crates.
This results to a huge file ~900 lines, witch is bound to get bigger.
I could split it into write and read operations or maybe there is
a better approach, but it's an issue that has to be addressed before any real
work can be done regarding the hierarchy features (move/delete).
I got limited time available until the end of the month (summer job to pay the bills), and in the two
weeks left, I would like to have the filter search work, at least at an
experimental stage. I would also like to figure things out with
and move on to the recursion stuff.
With August approaching I'm confident the nested crates feature will be completed in time.
By the way, I've found this it might help in the weeks to come.
July 17 - July 24
This week the focus was on refactoring
CrateStorage to make it easier to maintain and
read through the code, with classes that (I hope) make more sense.
The changes made are described here but I'll copy-paste them here for better readability.
CrateStorage was refactored into 5 new files:
The biggest challenge of course was the design of the architecture. I settled in the classes above by trying to keep each separate crate related data access to the DB to it's own class. When I read through the code I think it's easier to understand what does what.
July 24 - July 31
This week I implemeted a couple of new features for the nested crates.
Finally I've implemented crate filter using the crate
witch enables the recursive view of the crates.
Also just like
tracksFeature selecting the crate will invoke the
corresponding filter search and that's how the crate is displayed.
The whole process seems to me a bit clumsy but it works for now. I am thinking of ways I can improve that.
Duplicating a crate now makes a duplicate at level 1, containing just the tracks of this crate.
Here I optimised the way the crate naming works to secure the database from corrupted entries (same name in different crate, hierarchy naming conventions, etc)
Also you can freely rename crates while making sure the database won't break.
Also I fixed various small details and made little optimisations here and there.
Now about moving crates, I feel like it's a lot of effort for a feature not really that useful. The naming conventions will be really hard to enforce. Instead of moving crates you could just create them from scratch or rename them to what you want them to be.
I am thinking about being able to move only the level 1 crates so the workflow would be like: duplicate > move to position but it might be a bit confusing to the end user.
July 31 - August 14
I re implemented the crate filter and it no longer works with paths. It now uses the closure table. I found the new one to be a bit faster and it will also be much more stable in it's results because it doesn't rely on a path string like before and uses the closure table to get it's results.
I also changed the naming conventions according to what we think the user will expect. You can name a crate whatever you want except:
So it's basically more like a file system but you can't do this:
You can do this however (if you feel like it):
I added the crate summary that existed before to give some info for each crate.
Also if you rightclick a track and then add to crate, it now displays the crate path so the user can distinguish one crate from the other.
The feature works great so far, and I can't make it crash mixxx or slow the performance. Even with 15k tracks the crate queries take 100-150ms with the setup of ~50 crates.
August 14 - August 28
These last weeks I've been cleaning the code here and there, while I've also added some new features.
Moving crates throughout the tree is now possible via a right click menu, witch displays only the valid positions you can move the selected subtree. I found this sometime ago, witch provided me with a really god way of moving subtrees in a closure table via sql queries, and it proved to be real easy to implement, after I figured out how to do it in sqlite instead of mysql.
While moving crates I encountered the problem of a user moving a crate to a parent with the same name. Instead of throwing an error I decided to just add a “_” to the end of the crate's name. I made it so that no matter what you can't break the naming conventions.
I also corrected the failing crate filter tests.