Repository navigation
PySpeechModule - #1120
PySpeechModule#1120jsett wants to merge 15 commits into
Conversation
Better not take the burden of renaming, indeed :) My pypi account login is |
| :name: speechd-conf-adv | ||
|
|
||
| Timeout 0 | ||
| AddModule "kokoropy" "/home/user/src/kokoro_speechd_module/kokoro_module.py" "kokoropy.conf" |
There was a problem hiding this comment.
Mmm, no, modifying the speechd.conf configuration file is not the now-recommended way of adding a speechd module. One should rather just put it in the right place, which can be /usr/libexec/speech-dispatcher-modules/sd_kokoro and its configuration in /etc/speech-dispatcher/modules/kokoro.conf for system-wide installation, or ~/.local/libexec/speech-dispatcher-modules/sd_kokoro and ~/.config/speech-dispatcher/modules/kokoro.conf for per-user installation.
There was a problem hiding this comment.
I did not realize you could just drop scripts in to "/usr/libexec/speech-dispatcher-modules". It seemed like AddModule was the way of doing it. Its probably best to include both the system wide and local way in the doc's.
There was a problem hiding this comment.
hmm, if we place scripts into one of those directories and they are using a venv. Then we end up with a venv directory just hanging out in a random empty looking directory. I could see people deleting that venv directory and then breaking their script. I am not sure if placing a venv in ~/.local/libexec/speech-dispatcher-modules would be the right move either.
There was a problem hiding this comment.
I'm not sure to understand the problem with the venv?
I mean: if somebody is installing a module by hand, they would set up their venv somewhere, and put in ~/.local/libexec/speech-dispatcher-modules a script that use it through e.g. #!/home/user/src/kokoro_speechd_module/venv/bin/python . Would they really remove that venv directory? it's not empty, and we can make the documentation call the directory venv-for-synth-kokoro to make it explicit that it's needed
There was a problem hiding this comment.
Actually, I guess this is fine. I updated the doc's for it.
| :name: speechd-conf-quick | ||
|
|
||
| Timeout 0 | ||
| AddModule "dummypy" "/home/user/src/my_first_speechd_module/dummy.py" "dummypy.conf" |
There was a problem hiding this comment.
As mentioned above, better document the now-recommended "ship two files" approach.
| logging.debug(f"Ahead: {self._ahead}") | ||
| self.action_queue.put({"command": "speak", "val": item}) | ||
| # sleep if we have too much audio generated, only do this is max_ahead and min_ahead are set. | ||
| if (hasattr(self, "max_ahead") and hasattr(self, "min_ahead")): |
There was a problem hiding this comment.
Is this "ahead" management really needed?
My concern is that it was requested that speech-dispatcher not only be able to play the speech, but also allow some clients to retrieve the generated speech, and in that case we do not want to wait.
There was a problem hiding this comment.
This is actually a feature I would like to keep.
also allow some clients to retrieve the generated speech
I assume your talking about sending index mark, which it does to at the end of _send_audio in speechd_server.py, and this should line up with what has been just spoken.
Let me try and explain the problem this is solving with some charts..
The simplest way of playing+generating audio would look like this
gantt
dateFormat mm:ss
axisFormat %M:%S
Start : vert, v1, 00:00, 0m
Generate Audio : 30s
Play Audio: 30s
Generate Audio: 30s
Play Audio: 30s
Generate Audio: 30s
Play Audio: 30s
End: vert, v2, 03:00, 0m
This way, while simple to understand has major problems. The big one is that while your playing audio your computer is essentially doing nothing. Then when you need to generate audio, you have to sit there and wait for a few seconds before playing the next chunk of audio.
A better way would look like the following.
gantt
dateFormat mm:ss
axisFormat %M:%S
Start : vert, v1, 00:00, 0m
Generate Audio : 20s
Play Audio: 30s
Generate Audio: 00:20, 15s
Generate Audio: 00:35, 25s
Play Audio: 00:50, 25s
Generate Audio: 01:00, 15s
Play Audio: 01:15, 30s
Play Audio: 01:45, 30s
End: vert, v2, 02:15, 0m
This has its own disadvantage. For example after some time the generate audio buffer can get way ahead of your play audio and then your basically running your cpu really hard needlessly.
The best approach in my opinion would look something like this.
gantt
dateFormat mm:ss
axisFormat %M:%S
Start : vert, v1, 00:00, 0m
Generate Audio : 20s
Play Audio: 30s
Generate Audio: 00:20, 15s
Generate Audio: 00:35, 10s
Generate Audio: 00:45, 15s
Play Audio: 00:50, 25s
sleep while audio buffer is played down: 01:00, 60s
Play Audio: 01:15, 30s
Play Audio: 01:45, 30s
Generate Audio: 02:00, 15s
Play Audio: 02:15, 30s
End: vert, v2, 02:45, 0m
This way keeps track of how much audio it has played along with how long it took to generate that audio. Then when you reach a certain threshold it can sleep while the audio buffer is played down. Once the audio buffer is below the threshold again it can start generating audio again.
This last approach is the one we take if self.max_ahead and self.min_ahead are set. If they are not set then the second approach is taken.
There was a problem hiding this comment.
also allow some clients to retrieve the generated speech
I assume your talking about sending index mark
No, I am talking about #348
There was a problem hiding this comment.
The best approach in my opinion would look something like this
Ok, but the throttling should really not be handled by modules themselves, there is no reason not to factorize this into the speech dispatcher core, to save modules from having to manage this themselves. The core does know what is the actual progress of audio, and notably people have been asking for options to boost the playing speed in general, so the module can't even be sure at what speed the audio will be played.
We can just make the core stop reading the pipe when it has enough data, which will naturally block the synthesis module. If that poses problems, we can extend the protocol to let the server have some flow control, but really the modules should not be second-guessing what the server is doing with the audio content.
That is an issue really separate from having python modules, I have made an independent issue about it
#1132
There was a problem hiding this comment.
no reason not to factorize this into the speech dispatcher core
I hope this happens in the future.
There was a problem hiding this comment.
I think what I will do is give an option to the module developers to completely disable it if they choose. But I think its best for the time being to keep it as a feature as it makes deep learning models work well. As for the downloading, I think it would be wise to extend the protocol when that is implemented, to tell the module that its currently handling a download. I could see cases where downloading might want to be limited for example if the developer was using a proprietary api endpoint. Also module developers will likely want to handle downloads different then general speak commands. But downloading is outside the scope of this PR, and was not a feature I was working on here. With regards to changing the audio speed server side, I think this is actually a really really bad idea. You would likely get artifacts like really high pitch voice when speeding up or a really low deep voice when slowing down. I can't imaging that is really what anyone wants. I think its best to let the model handle that on the module side, that way it can create something that sounds natural (which many models do well by default).
There was a problem hiding this comment.
I hope this happens in the future.
As mentioned, it is recorded in #1132
I think what I will do is give an option to the module developers to completely disable it if they choose
The problem is that once you have provided the feature, you can't remove it, and then it becomes hard to make people move into the right way of doing things.
I think its best for the time being to keep it as a feature as it makes deep learning models work well
I don't really see why the current pipeline doesn't work, except if you indeed feed a lot of text that takes a long to synthesize, but is that really common usage?
I could see cases where downloading might want to be limited for example if the developer was using a proprietary api endpoint
Yes, that would have to be handled.
Also module developers will likely want to handle downloads different then general speak commands
Why?
But downloading is outside the scope of this PR, and was not a feature I was working on here.
Sure, but we have to think long-term to avoid getting into dead-ends.
You would likely get artifacts like really high pitch voice when speeding up or a really low deep voice when slowing down
No, you have various libraries that do it properly. Most commonly, for speech synthesis, libsonic is used.
I can't imaging that is really what anyone wants.
Believe it or not, but this is really exactly what people actually do use.
I think its best to let the model handle that on the module side
I thought as well, but people have confirmed it: when you push synthesizers to very fast speeds, they tend to get distorted results because you get outside of their tested area, and get bad artifacts. While libsonic does know to speed speech up with avoiding artifacts.
that way it can create something that sounds natural
I am talking about use cases that really do not care about the sound being natural, but about being efficient to be able to work fast.
|
|
||
| .. code-block:: python | ||
|
|
||
| [{"mark": "<mark>", "text": "<text>"}] |
There was a problem hiding this comment.
But then this loses all ssml tags except the marks? This should be documented as being just a helper for syntheses that do not know about ssml but can manage prosody over several pieces.
Ideally the underlying synthesis should be just given the ssml without any pre-parsing, so it can interpret whatever the speechd client requested.
There was a problem hiding this comment.
This function is purely a helper function for the user, its not used anywhere in the code. And yes it only returns the text and mark, that was the point of it.
the underlying synthesis should be just given the ssml without any pre-parsing
This is exactly what happens.
There was a problem hiding this comment.
This function is purely a helper function for the user, its not used anywhere in the code
Sure, I understand this
the underlying synthesis should be just given the ssml without any pre-parsing
This is exactly what happens.
Not if a python module uses the helper.
I mean, helpers are not useful if they provide a behavior that will pose problem.
There was a problem hiding this comment.
I will try and document it a little better so that its behavior is understood. But it does what I had intended. A more complete SSLM parser can be added in the future.
| def speak(self, ssml): | ||
| try: | ||
| # parse the ssml into chunk of text each with a mark. | ||
| for item in parse_ssml(ssml): |
There was a problem hiding this comment.
Mmm, but then we give only pieces of text to the underlying synthesis?
The issue is that screen readers which want fine-grain marking for on-screen speech progression will e.g. put marks between each word. Each word is then fed to the underlying synthesis separately, which can make the synthesis not produce any prosody at all, which will degrade the quality a lot.
I'd say better document using strip_ssml, which will by default bring good prosody, and add documentation to explain about managing marks vs supporting prosody. Some synthesis may be fed with several pieces with global prosody management, some others not, it really depends on the synth. Ideally synths would be fed with the complete ssml...
There was a problem hiding this comment.
put marks between each word. Each word is then fed to the underlying synthesis separately, which can make the synthesis not produce any prosody at all, which will degrade the quality a lot.
I have not seen this happen in practice. Is there a reader/client i can test that does this. Also the user is not forced to use the parse_ssml function, they can use their own method to handle the ssml if they choose to.
There was a problem hiding this comment.
I have not seen this happen in practice. Is there a reader/client i can test that does this.
Orca does this for instance.
Also the user is not forced to use the parse_ssml function, they can use their own method to handle the ssml if they choose to
Please do not confuse "user" and "programmer". While programmers may do things various ways, end users will use the module for various usages, which will include the orca usage, which will do such kind of marking. And users won't know to modify the module to avoid the splitting.
There was a problem hiding this comment.
I'm a KDE user so this is a little annoying for me, but i will try and setup a vm with gnome to test orca.
Sorry for the confusion about terminology, I will try and avoid saying user when I'm talking about the module developers.
There was a problem hiding this comment.
orca is not specific to gnome at all, it can run fine with a kde environment.
daxmawal
left a comment
There was a problem hiding this comment.
I compared this with PR #1104.
The main overlap is SSML stripping. module_strip_ssml() and strip_ssml() share the same goal, but their behavior differs regarding incomplete tags and unknown entities. We could reuse the tests from #1104, add cases involving bytes, and retain the pySpeechModule implementation.
The dictionaries representing voices already include the SPDVoice fields: name, language, and variant. They seem suitable as-is.
Regarding audio, AudioTrack in #1104 stored the data along with fields describing the format. I left a comment asking whether bits and num_channels would still be useful, or if we wanted to stick with the current format: 16-bit little-endian mono.
PR #1104 also added constants for message types. These commands are recognized here, but their type is not passed to speak().
| root = ET.fromstring(ssml_string) | ||
| except ET.ParseError: | ||
| # Wrap fragments in a temporary root node to ensure valid parsing | ||
| root = ET.fromstring(f"<root>{ssml_string}</root>") |
There was a problem hiding this comment.
| root = ET.fromstring(f"<root>{ssml_string}</root>") | |
| if isinstance(ssml_string, bytes): | |
| wrapped = b"<root>" + ssml_string + b"</root>" | |
| else: | |
| wrapped = f"<root>{ssml_string}</root>" | |
| root = ET.fromstring(wrapped) |
strip_ssml(b"Hello") returns "b'Hello'" instead of "Hello". The suggestion fixes that.
The tests from #1104 also cover incomplete tags and unknown entities, which now raise ParseError. Could we reuse those tests, add bytes cases, and decide whether to keep the old behaviour ?
There was a problem hiding this comment.
it'd probably be better to cope with parsing errors indeed. Better produce some audio than just an error. For blind people, that can make a difference between some oddities and a completely useless machine.
There was a problem hiding this comment.
I added these changes in yesterday.
| if (line == "SPEAK\n"): | ||
| self._speak() | ||
| elif (line == "SOUND_ICON\n"): | ||
| self._speak() | ||
| elif (line == "CHAR\n"): | ||
| self._speak() | ||
| elif (line == "KEY\n"): | ||
| self._speak() |
There was a problem hiding this comment.
Would it be useful to pass the message type to speak() too ? #1104 added constants for these types, and passing the type would let modules handle characters, keys and sound icons separately.
There was a problem hiding this comment.
Yes, we want to pass the message type, just like in the C API.
There was a problem hiding this comment.
The SOUND_ICON, CHAR, KEY were not really implemented fully. Ideally these would get passed along to their own specialized callbacks and the user could handle them appropriately. This would be a good feature for someone to implement in a future PR. But it was not something I was intending to include in this one. The core SPEAK command was what I was most concerned with. As such the lines 361-366 are mostly place holders for now.
There was a problem hiding this comment.
I am interested in handling this feature once this PR has been merged.
| # anything else is a fatal error. | ||
| raise Exception("Audio must be list or ndarray.") | ||
|
|
||
| stdout.write(f"705-bits=16\n") |
There was a problem hiding this comment.
I was wondering if the bits and num_channels fields from #1104's AudioTrack could still be useful here.
There was a problem hiding this comment.
We do want to be able to let the module specify the encoding, indeed. I'd rather not have a default value, better make sure that module writers know they have to specify it properly.
There was a problem hiding this comment.
Is it possible to specify the encoding to speechd? I did not think it was possible. That was why I had to include code in the Kokoro example to convert audio to the need format.
There was a problem hiding this comment.
Is it possible to specify the encoding to speechd?
Sure, that's the whole point of that write. The server core will then pass that on to the underlying audio library, which will know best how to re-encode it or not (depending whether the hardware can directly support the format or not). That allows syntheses to just emit the format they prefer.
There was a problem hiding this comment.
Yeah... I don't see that in the code
Lines 1230 to 1247 in b3ed835
There was a problem hiding this comment.
There must be a misunderstanding here.
You can't throw PCM_16
What do you mean by PCM_16?
All I'm saying is that the server wants raw audio samples, and they can be 16/8 bit, big/little endian, 11/22/44/48KHz, 1/2 chans, just like the synth prefers, it will be up to the server to convert as needed.
You can set 705-bits/705-num_channels/705-sample_rate/705-num_samples/705-big_endian, but thats it.
Yes, that is it, and that is all that I am saying.
add the bit's and num channels, as optional keys in the yield statement
No, as I mentioned these should be mandatory, so people writing module actually pay attention to that, it's not an optional thing.
I think we need to keep the formatting stuff.
It depends what you mean by "formatting" exactly.
…n catch errors and report them back to the server
…ispatcher-modules and /usr/libexec/speech-dispatcher-modules as the location for modules.
|
I have been thinking and decided that pySpeechModule is not mature enough to be included in speechd at this time. Good points about maintainability and lack of certain features have been made here. I will continue to look at ways of dealing with those issues in the future. As such I will be closing this PR, and keep pySpeechModule as an unofficial project outside of speechd. I would like to thank everyone for the time they placed into reviewing this PR. |
This a framework for creating speechd modules in python, along with documentation for using it, and a github workflow for building/publishing to PyPi.
Additional notes: For PyPi we can either recreate with a new name, or I think I can move it to a new owner(which I need a user name for the new owner). If we recreate it a number of different places will need to be updated with the new name. We will also need to change the trusted publishing repo for PyPi so that github can publish. For readthedocs I did not see a easy way to move it. It will almost certainly need to be recreated with a new name and then the code will need to be updated with that name.