"It was so difficult for me to speak" is correct grammatically, but the meaning is different from "It was too difficult for me to speak".
"It was so difficult (for me) to speak." --> 話すのがとても難しかったです。
Such a sentence is mostly used in speech and not in written text. "so" doesn't particularly refer to anything said before or after, but it just emphasizes that it was very difficult, and it carries an emotional tone.
That make sense thank you☺️