When the brothers appeared before Yosef the pasuk says “Vayizkor Yosef Es Hachalomos…”; Yosef remembered the dreams and only then did he accuse the shevatim of being spies and put them in jail for three days. Why does the Pasuk need to preface the accusation with the fact that he remembered the dreams?
The Chasam Sofer answers this with another question. How did Yosef have the gall to hold his brothers in jail for three days while his father Yaakov was starving for food in Eretz Yisroel? He answers that Yosef knew that Yaakov still had food left. How? From his dream.
In the first dream which represented the first time the brothers would bow down to him, he dreamed that each had a bundle of wheat and his was in the middle. He understood that this meant that they still had wheat in the house but nevertheless came to get more food from him.
Therefore only after he remembered the dream did he accuse them of being spies. He then kept them in jail three days knowing that there was still food in the house.