Rendered at 20:34:14 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
smoores 9 hours ago [-]
I took a week of from work recently to reimplement Storyteller's forced alignment algorithm. Storyteller[1] is an open source, self hosted platform for creating, managing, and reading/listening to "readaloud" books — books that have audiobook narration built in and can highlight each sentence (and/or word, with this new algorithm!) as it's read aloud. Forced alignment is the process of determining where each piece of text starts and ends in the audiobook.
Anyway, I am really pleased with how the new algorithm turned out! Hopefully someone else finds it interesting as well.
Wow I didn't even think about just using on device AI.
I'm going to set this one up and try it with kokoro which has the most natural for small size that I've seen. Wonder if the paperwhite 12th Gen can handle it.
And also setup the main repo for when I have the real audio book. Thanks!
leobg 46 minutes ago [-]
I tried running Kokoro on my iPhone XS (several years old), and it was slower than real-time. So, i wouldn’t expect it to be usable on a Kindle. But if you find a solution, please let me know.
orsenthil 7 hours ago [-]
This is excellent! I wanted this. I had noticed that Kindle had limited feature like this for certain books that support audiobooks. I had tried this style of reading with audiobook and ebook without a any automated sync, but hadn't pursued it. This looks cool. :)
smoores 7 hours ago [-]
Thanks! We're improving it all the time — we're working on big new releases ("v3") for the web app and mobile apps. There's a Discord server linked in the docs if you ever need any help out want to chat!
cainxinth 5 hours ago [-]
This is how I proofread. I play a screen reader while I read my work. The brain does some autocorrecting that you don’t even notice and occasionally skips over a typo when you read it without the audio.
mobeets 7 hours ago [-]
This is really impressive! One thing I’ve often wanted is a way to listen to an ebook (say, while cooking), and then later be able to sit down and read from where I left off. This essentially requires the same functionality that Storyteller offers (syncing progress through an ebook and audiobook), though arguably overkill for this use case (since I could handle a more coarse alignment). In any case, is this something Storyteller would be good for? Or are there other tools for the job?
smoores 7 hours ago [-]
That is actually the exact use case that I originally built Storyteller for! I wanted to listen to my books on long runs, and then switch to reading when I got back.
The Storyteller mobile apps have great support for this, I think. They have fully fledged audiobook players and ebook readers, and you can switch between the two with one tap. And then of course you can also double tap on a sentence and start playing from there, with the app highlighting the currently read sentence.
Xerox9213 7 hours ago [-]
What about reading on a koreader? Is bookbridge still the best bet?
Storyteller works great for this. It even syncs your position across devices. So when I really want to settle in to read, I go ahead and switch from my iPhone where I was listening to my Boox e-ink reader which syncs right to where I left off.
nottorp 1 hours ago [-]
I thought reading becomes immersive if you like the book...
smoores 26 minutes ago [-]
Sure! Some people have print disabilities like dyslexia and neurodivergence that makes reading text for an entire novel-length book challenging. Other people are perfectly competent print readers, but find that they enjoy having their book read to them. Audiobooks are their own art form, and it's nice to be able to enjoy them alongside text.
"Immersive reading" seems to be the industry term for this feature — I used it here because I though it was most likely to be recognized by a wide audience. Personally, and within the Storyteller ecosystem, I call it "readaloud," which I think is at least a little bit more useful of a phrase.
thelisu 4 hours ago [-]
This highlights the core reason why I don't like audiobooks, podcasts, or even movies, compared to reading: it's too slow. Looking at the example, I read two or three times as fast as the audio goes.
What I would need is the opposite: some eye-tracking software that plays the audio at the point I am reading. I don't know if it would be any useful though, as it would be very sped-up, but maybe it would increase "immersion" nevertheless.
smoores 4 hours ago [-]
Almost everyone who uses the readaloud features in Storyteller speeds up the audio playback quite a bit, often 2-3x, so that it matches their reading speed better. The Storyteller apps let you set different playback speeds for pure audio playback vs readaloud, since being able to see the text usually makes it easier to listen faster!
I supposed I should add: it's also, obviously, fine if this just isn't for you! Lots of people get a lot of value out of audiobooks (e.g., I really like to listen to audiobooks in the car or while running), but like to switch to reading when it's an option. And lots of people find readaloud super valuable, whether because they have a print disability or neurodivergence that makes reading challenging, or just because they like the experience of being read to. But lots of people are in neither of those groups!
ek_erilaz 2 hours ago [-]
I'd never considered such a tool before today, but the example sparked the exact same idea for me, as if it were the most obvious next step to invert the causal direction and externalize the inner reading voice (instead of trying to force the internal reading to follow the external audio). I'm not sure that it would improve reading comprehension or make the experience better in some ineffable way, but it would be really interesting to try it out. Questions arise:
- Would it help with reading comprehension? Speed? Enjoyment?
- If the audio was your own synthesized voice, would that help comprehension and immersion?
- If the audio was in a voice very different from your own, and you used this system for several hours a day, would it change your natural inner reading voice, or even some other aspect of your self-perception? I don't believe I've experienced Such a tight coupling between thoughts driving external responses before; I didn't expect to do so until real-time brain scanning got a lot better than current SotA.
- While reading, I frequently speed along and then jump back a word, sentence, or paragraph to slowly review it. In this system, the audio would slow down as I read back through more slowly, almost sounding out words along with me. I just read a book filled with the names of obscure french villages, and I couldn't just ignore them - instead bogged down in sounding them out and trying to guess how to pronounce them. Having a system sounding them out with/for me would have been awesome. This assumes the system is good at pronouncing things in various languages (mostly solved problem), and picking sane defaults when pronunciation conflicts arise (mostly unsolved?). In those cases, how hard would inferring be? Google translate is pretty good at generating localized pronunciations, but TTS systems struggle.
- Specifically for non-technical reading, it would be interesting to try having a very low level of ambient noise backgrounding the speech, driven by an LLM analyzing the text for obvious cues. Tavern sounds, rushing of wind in the mountains, crackling campfire, or even music generation (creepy dungeon etc etc)... When you pause reading for a moment to look away from the book, the speech obviously stops - but the tavern sounds continue. Might help people 'lock in' who normally get distracted from whatever they're reading.
altmanaltman 4 hours ago [-]
I read very fast and I noticed that after a certain point, it helps to not have the inner voice. Instead its more like I see a bunch of text and I understand it instead of going word by word. I don't think its possible to do this with audio.
lstodd 1 hours ago [-]
If that's about reading why do I need to zoom to 140 in FF to just begin reading?
smoores 1 hours ago [-]
Hm, I don't know what you're referring to. Is the text too small for you? I wrote the blog post on FF for Linux and FF for Android, and I didn't need to zoom in to review it, though some of the graphics do definitely end up with pretty small text on mobile.
usern20260720 6 hours ago [-]
I generated an immersive reading for Athenaze in a few afternoons with Claude
Anyway, I am really pleased with how the new algorithm turned out! Hopefully someone else finds it interesting as well.
[1] https://storyteller-platform.dev
I couldn't find it exactly in the docs. Sorry if I missed it.
If you align these and then pull onto koreader via opds or other way, will audio work over Bluetooth there too? Along with the highlighting.
I'm going to set this one up and try it with kokoro which has the most natural for small size that I've seen. Wonder if the paperwhite 12th Gen can handle it.
And also setup the main repo for when I have the real audio book. Thanks!
The Storyteller mobile apps have great support for this, I think. They have fully fledged audiobook players and ebook readers, and you can switch between the two with one tap. And then of course you can also double tap on a sentence and start playing from there, with the app highlighting the currently read sentence.
"Immersive reading" seems to be the industry term for this feature — I used it here because I though it was most likely to be recognized by a wide audience. Personally, and within the Storyteller ecosystem, I call it "readaloud," which I think is at least a little bit more useful of a phrase.
What I would need is the opposite: some eye-tracking software that plays the audio at the point I am reading. I don't know if it would be any useful though, as it would be very sped-up, but maybe it would increase "immersion" nevertheless.
I supposed I should add: it's also, obviously, fine if this just isn't for you! Lots of people get a lot of value out of audiobooks (e.g., I really like to listen to audiobooks in the car or while running), but like to switch to reading when it's an option. And lots of people find readaloud super valuable, whether because they have a print disability or neurodivergence that makes reading challenging, or just because they like the experience of being read to. But lots of people are in neither of those groups!
- Would it help with reading comprehension? Speed? Enjoyment?
- If the audio was your own synthesized voice, would that help comprehension and immersion?
- If the audio was in a voice very different from your own, and you used this system for several hours a day, would it change your natural inner reading voice, or even some other aspect of your self-perception? I don't believe I've experienced Such a tight coupling between thoughts driving external responses before; I didn't expect to do so until real-time brain scanning got a lot better than current SotA.
- While reading, I frequently speed along and then jump back a word, sentence, or paragraph to slowly review it. In this system, the audio would slow down as I read back through more slowly, almost sounding out words along with me. I just read a book filled with the names of obscure french villages, and I couldn't just ignore them - instead bogged down in sounding them out and trying to guess how to pronounce them. Having a system sounding them out with/for me would have been awesome. This assumes the system is good at pronouncing things in various languages (mostly solved problem), and picking sane defaults when pronunciation conflicts arise (mostly unsolved?). In those cases, how hard would inferring be? Google translate is pretty good at generating localized pronunciations, but TTS systems struggle.
- Specifically for non-technical reading, it would be interesting to try having a very low level of ambient noise backgrounding the speech, driven by an LLM analyzing the text for obvious cues. Tavern sounds, rushing of wind in the mountains, crackling campfire, or even music generation (creepy dungeon etc etc)... When you pause reading for a moment to look away from the book, the speech obviously stops - but the tavern sounds continue. Might help people 'lock in' who normally get distracted from whatever they're reading.