Ronanki: GSoC 2012 Pronunciation Evaluation Week 10

This week, I explored CART models a little bit, but couldn't complete it. The models are trained using wagon in speech tools with the following contextual information:

Current phone, previous phone, next phone, syllable postion, phonological features, phone type etc.,

Complete list of features are listed in the below URL:

http://talknicer.net/~ronanki/dur.desc

Once the training is completed, the output is built in a tree which is given as input along with testing data to wagon_test in speech tools and there by it predicts the duration of the each phone using the contextual information using tree structure.

Regarding replacing of traditional MFCC features either with PNCC or phonological features, I need to compute acoustic models for WSJ database replacing these features instead of MFCC. It's in process, and once acoustic models are built, the rest of the testing process is same. 

Work to do:

By next week, I would be able to complete one of these two and the next one thereafter. In the final week, I upload all codes to svn and integrate these new techniques with the current working pronunciation evaluation model @ http://talknicer.net/~ronanki/

 

Ronanki: GSoC 2012 Pronunciation Evaluation week 9

This week, I finished with my random phrase pronunciation evaluation and is in testing phase @ http://talknicer.net/~ronanki/test/index.html

The system can provide evaluation scoring for any random sentence. It also gives feedback for mispronunciation and rate of duration at word level. Please, test the system and mail me the bugs if any. Please avoid giving proper nouns and punctuation marks while testing the system.

For doing this, I evaluated entire TIMIT dataset  and the statistics for each phone are evaluated at three positions: 

Begin/Middle/End (0/1/2). The count in the last column represents the number of times each phone occurred at each position. The statistics are @ http://talknicer.net/~ronanki/phrase_data/statistics/TIMIT_statistics.txt

Next week, I am going to implement CART models so that each phone can be compared with respective phone in better context. Regarding features, I studied about Power Normalized Cepstral Coefficients (PNCC) which are more robust towards speech recognition even in noisy environment. PNCC are 13 in dimension, computationally more cost than MFCC but performs better than MFCC in speech recognition. I downloaded the available matlab code @ http://www.cs.cmu.edu/~robust/archive/algorithms/PNCC_IEEETran/ and trying some experiments on nTIMIT database. I also implemented phonological mapping with current state of spectral features (MFCC) using ANN. Currently, I am in testing phase of speech recognition using all these features. 

Ronanki: GSoC 2012 Pronunciation Evaluation week 8

This week, I mainly concentrated on integrating everything with web @
http://talknicer.net/~ronanki/test/

The following are the ones which are integrated:

1. File upload option with different formats (wav/wma/mp3) is provided.
2. All test cases are evaluated while recording and it allows only those recordings which are near to the perfect case.
3. The calculate score button provides the feedback page @ http://talknicer.net/~ronanki/test/scores_page.html (still some of the columns in UI are under construction)
4. Phrase entry as per user's choice and then score calculation page @ http://talknicer.net/~ronanki/test/random.html is also under construction.
5. As of now, statistics for random phrase entry as per user's choice are derived from TIMIT database which covers 630 speakers with 10 recordings from each one.

Next Tasks:

1. Feature extraction (Power-Normalized Cepstral Coefficients and phonological features)
2. CART models (for efficient score calculation in random phrase method based on contextual information)

Regarding under construction pages, the back-end codes were developed and uploaded at sourceforge. Only, the web pages need to be build dynamically. Will be done in parallel with the current next tasks.

Ronanki: GSoC 2012 Pronunciation Evaluation week 7

Last week, I continued to work on spectral features and phonological features and their mapping based on neural network training for first few days. Based on forced-alignment/manual labels if exists, these phonological features http://talknicer.net/~ronanki/phonological_features/feature_stream for each phone in a phrase are repeated against it's spectral features. I am looking over CSLU Toolkit which uses a neural net for feature-to-diphone decoding and stopped at that point to work out later after mid-evaluation.

Later, I worked on integration of acoustic/duration scores along with edit-distance grammar decoding with the current website for exemplar outlier analysis.

I tried with many test cases such as
1. silence
2. noisy speech
3. Junk speech
4. Random sentence
5. Actual sentence shortened in the end
6. Actual sentence skipped the beginning

In test cases from 1-6, the forced alignment did not reach the final state and failed to create phone segmentation file, label file which contains acoustic scores, phone labels respectively.

7. Actual sentence
8. Actual sentence with more silence both at beginning and end
9. Actual sentence with one small word skip in the middle of phrase
10. Similar sounding sentences such as

Ex. Utterance: 

Approach the teaching of pronunciation with more confidence

Tested Similar sounding: 

a. Approach the teaching opponents the nation with over confidence

b. Approach the preaching opponents the nation with confidence

In test cases from 7-10, the forced alignment worked and generated acoustic scores, phone labels. Then, I moved on to edit-distance grammar decoding testing accuracy on cases 7-10 so that I can set a threshold parameter to distinguish between cases (7,8,9) and (10)

Earlier, I tested for cases 7,8 with phrase decoder in edit-distance and reported it around 73% and the accuracy is < 40% for case 10 so that I can easily set the threshold parameter such as accuracy = x>0.4 ? T : F

I also discussed with my mentor James Salsman on giving weights to words based on parts of speech for phrase output score and here is what he proposed and all the units are in db which represents relative loudness in English.

(%wt, %pos); # scoring weights and names of parts of speech

$wt{'q'} = 1.0; $pos{'q'} = 'quantifier';
$wt{'n'} = 0.9; $pos{'n'} = 'noun';
$wt{'v'} = 0.9; $pos{'v'} = 'verb';
$wt{'-'} = 0.8; $pos{'-'} = 'negative';
$wt{'w'} = 0.8; $pos{'w'} = 'adverb';
$wt{'m'} = 0.8; $pos{'m'} = 'adjective';
$wt{'o'} = 0.7; $pos{'o'} = 'pronoun';
$wt{'s'} = 0.6; $pos{'s'} = 'possessive';
$wt{'p'} = 0.6; $pos{'p'} = 'preposition';
$wt{'c'} = 0.5; $pos{'c'} = 'conjunction';
$wt{'a'} = 0.4; $pos{'a'} = 'article';

Hope, I do it and launch the site after integrating everything before mid-evaluation submission.

Ronanki: GSoC 2012 Pronunciation Evaluation Week 6

I uploaded all my codes (except few ongoing) here at 

 http://cmusphinx.svn.sourceforge.net/viewvc/cmusphinx/branches/speecheval/ronanki/scripts/ . Please follow README files in each folder for detailed instructions on how to use them. 

This week, I have concentrated on new features for speech recognition. I read a paper on Power-Normalized Cepstral Coefficients [1] which are more robust towards speech recognition and a few papers on phonological features [2],[3]. I hope to investigate mapping the acoustic speech features of each phoneme derived from machine phonetic transcription to phonological features. Using this mapping, mispronunciations at phone level can be identified using phonological features along with acoustic pronunciation scores and edit distances. I got some mapping here at http://talknicer.net/~ronanki/phonological_features/ based on those papers.

Ongoing tasks:

1. In random phrase scoring method, another column is added to store the position of each phone with respect to word (begin/middle/end) such that each phone will have three statistics 
http://talknicer.net/~ronanki/phrase_data/all_phrases_stats_position

2. Standard word scores are derived along with phoneme standard (acoustic + duration) scores in the current forced-alignment.

3. Linking edit-distance algorithm with pronunciation evaluation website

4. Complete a full-pledged website at http://talknicer.net/~ronanki/test/ with all test cases (junk speech, silence, misread etc.,) before mid-evaluation and publicize the system so that it can be tested by large number of users. 

References:

[1] Chanwoo Kim and Richard M.Stern, "Power-Normalized Cepstral Coefficients (PNCC) for Robust Speech Recognition", ICASSP 2012.

http://www.cs.cmu.edu/~chanwook/MyPapers/ICASSP2012_PNCC_ver09.pdf

[2] Katrin Kirchhoff a, Gernot A. Fink b, Gerhard Sagerer b, "Combining acoustic and articulatory feature information for robust speech recognition", Speech Communications 37 (2002) 303–319.

http://www.sciencedirect.com/science/article/pii/S0167639301000206

[3]  S. King and P. Taylor, “Detection of phonological features in continuous speech using neural networks,” Computer Speech and Language, vol. 14, no. 4, pp. 333–353, 2000.

http://www.era.lib.ed.ac.uk/bitstream/1842/1001/1/King_Taylor_csl2000.pdf

Daily progress reports in comments here

Quick mentor note: We are converting to daily progress reports which I will combine into draft blog posts that the students will proofread, copy-edit, and approve for publication.  This will help keep all three of us on schedule. Sorry I am behind. The good news is that both students made it from "on schedule" to "ahead of schedule" in a sprint for the evaluations.

Congratulations, Troy and Ronanki!

Please post your daily-ish (4 or more per week) progress reports here. Thanks!

Troy: GSoC 2012 Pronunciation Evaluation Week 5

Sorry for the late update. The following are the things I did in Week 5; mainly problem solving.

1) Solving the Flash-based recorder update which prevented users from using their microphones. 

At the beginning, before the Flash player 11.2 and 11.3 update, the audio recorder I created using Flex worked fine. Users could simply right click the recorder and select the "Settings" to allow microphone access. However, with the new updates, that option is disabled without any error message. 

To solve this problem, people suggested adding websites into the online global privacy list. However, after trying many times that was still not working for the audio recorder. 

Furthermore, http://englishcentral.com/ which also uses Flash-based recording has a popup window from their recording button (a microphone image) with the Flash Microphone privacy setting dialogue. Checking the accessibility of microphone in code and prompting for the setting dialogue when necessary helps provide the solution:

First, checking whether the microphone is available, if not show the microphone list dialogue of Flash object ask the user to plugin a microphone:

var mic:Microphone = Microphone.getMicrophone();

if(!mic) {

Alert.show("No microphone available");

debug("No microphone available");

Security.showSettings("microphone");

}

Otherwise, check whether the microphone is accessible or not, if it is muted, prompt the privacy dialogue to ask user to allow the microphone access:

if(mic.muted) {

debug("Microphone muted!");

Security.showSettings("privacy");

}

With these testing during the initialization stage of the Flash recorder, it can allow users to enable the microphone access at the early beginning. One interesting thing is that after doing this, the "Setting" option of the Flash object now is clickable. 

Now, looking back to the code solving the problem, which is so apparent, however, before you know the answer, it is really hard to predict. 

2) Cross-browser Flash recorder compatibility

As the Flash recorder problem was solved as above, I was happy to update the source code in the trunk and our server and hoped to see the site working nicely. But the browser shows that the Flash recorder cannot load, the only information I got is "Error 2046"....

To try to solve this problem, I Googled a bunch of pages and tried several suggestions, the first which suggested I clear the browser cache and then set the Flash player to not save local cache and then re-enable its local cache (some kind of clear Flash player local cache), which gives some progress by changing "Error 2046" to "Error 2032". 

For "Error 2032", there are mainly two groups of explanations, one saying there is something wrong with the URLs in Actionscript's HTTPRequests, which seems unlikely because those URLs are definitely correct and are under the same folder as the player. The other is an RSL problem of the mxmlc Flash compiler. To solve the RSL linkage problem, go to the "Flex Build Path" properties page, "Library path" tab and change the framework linkage to "merged into code".

[Mentor note: Requesting compatibility with earlier versions of Flash ActionScript using compiler switches may or may not help here.]

3) Adding a password change page

4) Refining the user extra information update page to reflect the existing user information if available, instead of always showing the default values.

The website for exemplary recordings is now at a usable stage. 

In this week, I will try to accomplish these things:

1) Phrase data entry for administrators (with text, exemplar pronunciations, homograph disambiguation, phonemes, parts of speech per word, etc.;

2) Design recording prompts to start our exemplary recording data collection;

3) Bug fixing and system testing;

4) Study the Amazon Mechanical Turk and start thinking how to incorporate our speech data collection on to that platform.