Segue: how Collectoh listens to your music

Tags are good at facts. They'll tell you what year an album came out and that somebody once filed it under Rock. They're no help with the question people actually ask of a music collection, which is "what else sounds like this?" Two songs filed under Rock can be a lullaby and a riot. Two from different decades and different shelves can belong side by side on the same late-night playlist.

So Collectoh listens.

#Listening, on your own computer

Every track in your music libraries is heard by a small audio model, the listening half of an open model called CLAP, which Collectoh fetches once (about 80 MB) and runs on your server's own processor. It listens to three ten-second stretches of each track, a quarter, a half and three-quarters of the way through, and turns each into a description of how it sounds: not a genre or a mood word, but a position among all the sounds it has learned. The three are averaged into one, and that single description is what everything below is built on.

None of this leaves the server: not the music, not what the model makes of it, and not what you play. The listening happens in the background whenever there's something new to hear, and it steps aside whenever anyone is watching or listening to something. A scan queues its new tracks when it finishes, anything missed is caught overnight, and a file that won't decode is tried again a week later. On our development machine, a recent desktop processor, each track takes a little under a second, so a library of 16,000 tracks is heard in about four hours, and only once. Diagnostics shows how far it has got.

Alongside the model, a simpler nightly analysis measures each track's tempo, key and energy, which you'll see on the track's page. Those numbers stand in for any track the model hasn't heard yet, and for every track if you switch the listening off with Hear tracks for sonic similarity in Settings → Music.

#Sounds like this

Open a track and you'll find a shelf called Sounds like this: the tracks nearest to it in sound, drawn from the libraries you're allowed to see. To keep it a spread rather than the rest of the same record, no more than two come from any one album and no more than three from any one artist. If the model hasn't heard the track yet, the shelf is called More like this instead and goes by the nightly measurements, so you always know which you're getting.

#Radio and mixes, drawn around a sound

Radio starts from a track, an album or an artist. An album's sound is the average of its tracks, and an artist's the average of theirs, so an album radio begins from the album's overall character rather than whichever song happens to open it.

Collectoh then draws a mix around that sound. Each next track is picked at random from among the nearest, weighted to stay close to both where you started and the track just played, so the mix flows from one song to the next without wandering off. Anything you've played lately is held back, and so is the artist you've just heard. The result is a queue like any other: skip, reorder or add to it as you please.

#Segue

Radio stays near where it started. A segue goes somewhere.

Choose Segue… from a track's menu and Collectoh asks where you'd like to end up: search for any other track in your library and pick it. It then draws a line through the sound between the two and walks along it, taking at each step the nearest track it hasn't used yet, and preferring a different artist from the one before. You see the way there before you commit, numbered, with each track's artist beside it, and if you'd rather end up elsewhere, Somewhere else lets you choose again. Play the way there turns the path into an ordinary queue, a dozen tracks with both ends included.

It's a good way to get from a quiet piano piece to something with a lot more drums without a jolt in between, and a good way to find your way back to the middle of your own collection. If either end hasn't been heard yet, or the listening is switched off, Collectoh says so rather than guessing: unlike radio, a segue has nothing else to fall back on.

#What it doesn't do

• It doesn't send your music, what the model hears or what you play anywhere. There's no cloud service involved.
• It doesn't change your files or your tags. What it learns is kept in Collectoh's database.
• It doesn't write playlists with a chatbot. We thought about it and left it out: it would need a language model your server doesn't have, and a hosted one would mean sending your listening habits somewhere else.
• It doesn't reach past what you're allowed to see. Sounds like this, radio and segues draw only on the libraries your account can open.

#Nothing to switch on

Listening is on by default, as long as the database has the vector extension it needs, and the database in Collectoh's Docker setup includes it. If yours doesn't, Diagnostics says so and what to do, and nothing else is lost: the shelf and radio simply go by the nightly measurements until it's sorted. Installing Collectoh covers the database, and the docs say more about music libraries.

And when all you want is an album from start to finish, it still plays gaplessly, exactly as it was made.