Skip to content

PySpeechModule - #1120

Closed
jsett wants to merge 15 commits into
brailcom:masterfrom
jsett:pySpeechModule
Closed

jsett wants to merge 15 commits into
brailcom:masterfrom
jsett:pySpeechModule

Conversation

@jsett

@jsett jsett commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

This a framework for creating speechd modules in python, along with documentation for using it, and a github workflow for building/publishing to PyPi.

Additional notes: For PyPi we can either recreate with a new name, or I think I can move it to a new owner(which I need a user name for the new owner). If we recreate it a number of different places will need to be updated with the new name. We will also need to change the trusted publishing repo for PyPi so that github can publish. For readthedocs I did not see a easy way to move it. It will almost certainly need to be recreated with a new name and then the code will need to be updated with that name.

@sthibaul

sthibaul commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

For PyPi we can either recreate with a new name, or I think I can move it to a new owner(which I need a user name for the new owner).

Better not take the burden of renaming, indeed :)

My pypi account login is sthibaul

@sthibaul sthibaul left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this work!
This looks nice as a first implementation, we will be able to refine on top of this.
@daxmawal could you also have a look to check if what you had implemented also shows up here?

:name: speechd-conf-adv

Timeout 0
AddModule "kokoropy" "/home/user/src/kokoro_speechd_module/kokoro_module.py" "kokoropy.conf"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mmm, no, modifying the speechd.conf configuration file is not the now-recommended way of adding a speechd module. One should rather just put it in the right place, which can be /usr/libexec/speech-dispatcher-modules/sd_kokoro and its configuration in /etc/speech-dispatcher/modules/kokoro.conf for system-wide installation, or ~/.local/libexec/speech-dispatcher-modules/sd_kokoro and ~/.config/speech-dispatcher/modules/kokoro.conf for per-user installation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not realize you could just drop scripts in to "/usr/libexec/speech-dispatcher-modules". It seemed like AddModule was the way of doing it. Its probably best to include both the system wide and local way in the doc's.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, if we place scripts into one of those directories and they are using a venv. Then we end up with a venv directory just hanging out in a random empty looking directory. I could see people deleting that venv directory and then breaking their script. I am not sure if placing a venv in ~/.local/libexec/speech-dispatcher-modules would be the right move either.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure to understand the problem with the venv?

I mean: if somebody is installing a module by hand, they would set up their venv somewhere, and put in ~/.local/libexec/speech-dispatcher-modules a script that use it through e.g. #!/home/user/src/kokoro_speechd_module/venv/bin/python . Would they really remove that venv directory? it's not empty, and we can make the documentation call the directory venv-for-synth-kokoro to make it explicit that it's needed

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I guess this is fine. I updated the doc's for it.

:name: speechd-conf-quick

Timeout 0
AddModule "dummypy" "/home/user/src/my_first_speechd_module/dummy.py" "dummypy.conf"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned above, better document the now-recommended "ship two files" approach.

logging.debug(f"Ahead: {self._ahead}")
self.action_queue.put({"command": "speak", "val": item})
# sleep if we have too much audio generated, only do this is max_ahead and min_ahead are set.
if (hasattr(self, "max_ahead") and hasattr(self, "min_ahead")):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this "ahead" management really needed?

My concern is that it was requested that speech-dispatcher not only be able to play the speech, but also allow some clients to retrieve the generated speech, and in that case we do not want to wait.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually a feature I would like to keep.

also allow some clients to retrieve the generated speech

I assume your talking about sending index mark, which it does to at the end of _send_audio in speechd_server.py, and this should line up with what has been just spoken.

Let me try and explain the problem this is solving with some charts..

The simplest way of playing+generating audio would look like this

gantt

dateFormat mm:ss

axisFormat %M:%S

Start : vert, v1, 00:00, 0m

Generate Audio : 30s

Play Audio: 30s

Generate Audio: 30s

Play Audio: 30s

Generate Audio: 30s

Play Audio: 30s

End: vert, v2, 03:00, 0m

Loading

This way, while simple to understand has major problems. The big one is that while your playing audio your computer is essentially doing nothing. Then when you need to generate audio, you have to sit there and wait for a few seconds before playing the next chunk of audio.

A better way would look like the following.

gantt
    dateFormat mm:ss
    axisFormat %M:%S
    Start : vert, v1, 00:00, 0m
    Generate Audio : 20s
    Play Audio:  30s
    Generate Audio: 00:20, 15s
    Generate Audio: 00:35, 25s
    Play Audio: 00:50, 25s
    Generate Audio: 01:00, 15s
    Play Audio: 01:15, 30s
    Play Audio: 01:45, 30s
    End: vert, v2, 02:15, 0m
Loading

This has its own disadvantage. For example after some time the generate audio buffer can get way ahead of your play audio and then your basically running your cpu really hard needlessly.

The best approach in my opinion would look something like this.

gantt
    dateFormat mm:ss
    axisFormat %M:%S
    Start : vert, v1, 00:00, 0m
    Generate Audio : 20s
    Play Audio:  30s
    Generate Audio: 00:20, 15s
    Generate Audio: 00:35, 10s
    Generate Audio: 00:45, 15s
    Play Audio: 00:50, 25s
    sleep while audio buffer is played down: 01:00, 60s
    Play Audio: 01:15, 30s
    Play Audio: 01:45, 30s
    Generate Audio: 02:00, 15s
    Play Audio: 02:15, 30s
    End: vert, v2, 02:45, 0m
Loading

This way keeps track of how much audio it has played along with how long it took to generate that audio. Then when you reach a certain threshold it can sleep while the audio buffer is played down. Once the audio buffer is below the threshold again it can start generating audio again.

This last approach is the one we take if self.max_ahead and self.min_ahead are set. If they are not set then the second approach is taken.

@sthibaul sthibaul Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also allow some clients to retrieve the generated speech

I assume your talking about sending index mark

No, I am talking about #348

@sthibaul sthibaul Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The best approach in my opinion would look something like this

Ok, but the throttling should really not be handled by modules themselves, there is no reason not to factorize this into the speech dispatcher core, to save modules from having to manage this themselves. The core does know what is the actual progress of audio, and notably people have been asking for options to boost the playing speed in general, so the module can't even be sure at what speed the audio will be played.

We can just make the core stop reading the pipe when it has enough data, which will naturally block the synthesis module. If that poses problems, we can extend the protocol to let the server have some flow control, but really the modules should not be second-guessing what the server is doing with the audio content.

That is an issue really separate from having python modules, I have made an independent issue about it
#1132

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no reason not to factorize this into the speech dispatcher core

I hope this happens in the future.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think what I will do is give an option to the module developers to completely disable it if they choose. But I think its best for the time being to keep it as a feature as it makes deep learning models work well. As for the downloading, I think it would be wise to extend the protocol when that is implemented, to tell the module that its currently handling a download. I could see cases where downloading might want to be limited for example if the developer was using a proprietary api endpoint. Also module developers will likely want to handle downloads different then general speak commands. But downloading is outside the scope of this PR, and was not a feature I was working on here. With regards to changing the audio speed server side, I think this is actually a really really bad idea. You would likely get artifacts like really high pitch voice when speeding up or a really low deep voice when slowing down. I can't imaging that is really what anyone wants. I think its best to let the model handle that on the module side, that way it can create something that sounds natural (which many models do well by default).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hope this happens in the future.

As mentioned, it is recorded in #1132

I think what I will do is give an option to the module developers to completely disable it if they choose

The problem is that once you have provided the feature, you can't remove it, and then it becomes hard to make people move into the right way of doing things.

I think its best for the time being to keep it as a feature as it makes deep learning models work well

I don't really see why the current pipeline doesn't work, except if you indeed feed a lot of text that takes a long to synthesize, but is that really common usage?

I could see cases where downloading might want to be limited for example if the developer was using a proprietary api endpoint

Yes, that would have to be handled.

Also module developers will likely want to handle downloads different then general speak commands

Why?

But downloading is outside the scope of this PR, and was not a feature I was working on here.

Sure, but we have to think long-term to avoid getting into dead-ends.

You would likely get artifacts like really high pitch voice when speeding up or a really low deep voice when slowing down

No, you have various libraries that do it properly. Most commonly, for speech synthesis, libsonic is used.

I can't imaging that is really what anyone wants.

Believe it or not, but this is really exactly what people actually do use.

I think its best to let the model handle that on the module side

I thought as well, but people have confirmed it: when you push synthesizers to very fast speeds, they tend to get distorted results because you get outside of their tested area, and get bad artifacts. While libsonic does know to speed speech up with avoiding artifacts.

that way it can create something that sounds natural

I am talking about use cases that really do not care about the sound being natural, but about being efficient to be able to work fast.


.. code-block:: python

[{"mark": "<mark>", "text": "<text>"}]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But then this loses all ssml tags except the marks? This should be documented as being just a helper for syntheses that do not know about ssml but can manage prosody over several pieces.

Ideally the underlying synthesis should be just given the ssml without any pre-parsing, so it can interpret whatever the speechd client requested.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function is purely a helper function for the user, its not used anywhere in the code. And yes it only returns the text and mark, that was the point of it.

the underlying synthesis should be just given the ssml without any pre-parsing

This is exactly what happens.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function is purely a helper function for the user, its not used anywhere in the code

Sure, I understand this

the underlying synthesis should be just given the ssml without any pre-parsing

This is exactly what happens.

Not if a python module uses the helper.

I mean, helpers are not useful if they provide a behavior that will pose problem.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will try and document it a little better so that its behavior is understood. But it does what I had intended. A more complete SSLM parser can be added in the future.

def speak(self, ssml):
try:
# parse the ssml into chunk of text each with a mark.
for item in parse_ssml(ssml):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mmm, but then we give only pieces of text to the underlying synthesis?

The issue is that screen readers which want fine-grain marking for on-screen speech progression will e.g. put marks between each word. Each word is then fed to the underlying synthesis separately, which can make the synthesis not produce any prosody at all, which will degrade the quality a lot.

I'd say better document using strip_ssml, which will by default bring good prosody, and add documentation to explain about managing marks vs supporting prosody. Some synthesis may be fed with several pieces with global prosody management, some others not, it really depends on the synth. Ideally synths would be fed with the complete ssml...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

put marks between each word. Each word is then fed to the underlying synthesis separately, which can make the synthesis not produce any prosody at all, which will degrade the quality a lot.

I have not seen this happen in practice. Is there a reader/client i can test that does this. Also the user is not forced to use the parse_ssml function, they can use their own method to handle the ssml if they choose to.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not seen this happen in practice. Is there a reader/client i can test that does this.

Orca does this for instance.

Also the user is not forced to use the parse_ssml function, they can use their own method to handle the ssml if they choose to

Please do not confuse "user" and "programmer". While programmers may do things various ways, end users will use the module for various usages, which will include the orca usage, which will do such kind of marking. And users won't know to modify the module to avoid the splitting.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a KDE user so this is a little annoying for me, but i will try and setup a vm with gnome to test orca.

Sorry for the confusion about terminology, I will try and avoid saying user when I'm talking about the module developers.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

orca is not specific to gnome at all, it can run fine with a kde environment.

Comment thread README.md Resolved

@daxmawal daxmawal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I compared this with PR #1104.

The main overlap is SSML stripping. module_strip_ssml() and strip_ssml() share the same goal, but their behavior differs regarding incomplete tags and unknown entities. We could reuse the tests from #1104, add cases involving bytes, and retain the pySpeechModule implementation.

The dictionaries representing voices already include the SPDVoice fields: name, language, and variant. They seem suitable as-is.

Regarding audio, AudioTrack in #1104 stored the data along with fields describing the format. I left a comment asking whether bits and num_channels would still be useful, or if we wanted to stick with the current format: 16-bit little-endian mono.

PR #1104 also added constants for message types. These commands are recognized here, but their type is not passed to speak().

root = ET.fromstring(ssml_string)
except ET.ParseError:
# Wrap fragments in a temporary root node to ensure valid parsing
root = ET.fromstring(f"<root>{ssml_string}</root>")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
root = ET.fromstring(f"<root>{ssml_string}</root>")
if isinstance(ssml_string, bytes):
wrapped = b"<root>" + ssml_string + b"</root>"
else:
wrapped = f"<root>{ssml_string}</root>"
root = ET.fromstring(wrapped)

strip_ssml(b"Hello") returns "b'Hello'" instead of "Hello". The suggestion fixes that.

The tests from #1104 also cover incomplete tags and unknown entities, which now raise ParseError. Could we reuse those tests, add bytes cases, and decide whether to keep the old behaviour ?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it'd probably be better to cope with parsing errors indeed. Better produce some audio than just an error. For blind people, that can make a difference between some oddities and a completely useless machine.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added these changes in yesterday.

Comment on lines +359 to +366
if (line == "SPEAK\n"):
self._speak()
elif (line == "SOUND_ICON\n"):
self._speak()
elif (line == "CHAR\n"):
self._speak()
elif (line == "KEY\n"):
self._speak()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be useful to pass the message type to speak() too ? #1104 added constants for these types, and passing the type would let modules handle characters, keys and sound icons separately.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we want to pass the message type, just like in the C API.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SOUND_ICON, CHAR, KEY were not really implemented fully. Ideally these would get passed along to their own specialized callbacks and the user could handle them appropriately. This would be a good feature for someone to implement in a future PR. But it was not something I was intending to include in this one. The core SPEAK command was what I was most concerned with. As such the lines 361-366 are mostly place holders for now.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am interested in handling this feature once this PR has been merged.

# anything else is a fatal error.
raise Exception("Audio must be list or ndarray.")

stdout.write(f"705-bits=16\n")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was wondering if the bits and num_channels fields from #1104's AudioTrack could still be useful here.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do want to be able to let the module specify the encoding, indeed. I'd rather not have a default value, better make sure that module writers know they have to specify it properly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to specify the encoding to speechd? I did not think it was possible. That was why I had to include code in the Kokoro example to convert audio to the need format.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to specify the encoding to speechd?

Sure, that's the whole point of that write. The server core will then pass that on to the underlying audio library, which will know best how to re-encode it or not (depending whether the hardware can directly support the format or not). That allows syntheses to just emit the format they prefer.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah... I don't see that in the code

speechd/src/server/output.c

Lines 1230 to 1247 in b3ed835

if (strncmp(p, "705-AUDIO", strlen("705-AUDIO")) == 0 && p[strlen("705-AUDIO")] == '\0') {
p += strlen("705-AUDIO") + 1;
break;
}
if (strncmp(p, "705-big_endian=", strlen("705-big_endian=")) == 0) {
format = atoi(p + strlen("705-big_endian="));
}
#define SET_AUDIO_TRACK_PARAM(name) \
else if (strncmp(p, "705-"#name"=", strlen("705-"#name"=")) == 0) { \
track.name = atoi(p+4+strlen(#name)+1); \
MSG2(5, "output_module", \
"Got audio parameter "#name" %d", track.name); \
}
SET_AUDIO_TRACK_PARAM(bits)
SET_AUDIO_TRACK_PARAM(num_channels)
SET_AUDIO_TRACK_PARAM(sample_rate)
SET_AUDIO_TRACK_PARAM(num_samples)
You can set 705-bits/705-num_channels/705-sample_rate/705-num_samples/705-big_endian, but thats it. You can't throw PCM_16 at it and just expect it know what that is. I will add the bit's and num channels, as optional keys in the yield statement. But unless I am missing something I think we need to keep the formatting stuff.

@sthibaul sthibaul Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There must be a misunderstanding here.

You can't throw PCM_16

What do you mean by PCM_16?

All I'm saying is that the server wants raw audio samples, and they can be 16/8 bit, big/little endian, 11/22/44/48KHz, 1/2 chans, just like the synth prefers, it will be up to the server to convert as needed.

You can set 705-bits/705-num_channels/705-sample_rate/705-num_samples/705-big_endian, but thats it.

Yes, that is it, and that is all that I am saying.

add the bit's and num channels, as optional keys in the yield statement

No, as I mentioned these should be mandatory, so people writing module actually pay attention to that, it's not an optional thing.

I think we need to keep the formatting stuff.

It depends what you mean by "formatting" exactly.

@jsett

jsett commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

I have been thinking and decided that pySpeechModule is not mature enough to be included in speechd at this time. Good points about maintainability and lack of certain features have been made here. I will continue to look at ways of dealing with those issues in the future. As such I will be closing this PR, and keep pySpeechModule as an unofficial project outside of speechd. I would like to thank everyone for the time they placed into reviewing this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants