Finally I found a solution for the servo jitter problem today. As suspected, the SoftwareSerial library of Arduino was interfering with the EasyVR library.
After reconfiguring the software to use hardware serial now it is working as expected.
So what we have here is currently 4 channels (can be raised up to 13) that can be controlled by speech.
The test below confirms the concept (with 2 channels). For english language commands wind to 01:15:
Within the next few days I shall build this solution to the rover and find out how to make the control smooth. Stay tuned