It's quite natural to have to stop and think before responding when answering a question posed to you in a second language. Your brain needs time to "digest" the foreign phrase, translate it into your native language, and then you need to mentally compose your answer in the foreign language. Even when you're fluent in a second language, the process can be difficult - I used to deal regularly with folks from the Mexican division of an automotive company as part of my job. I remember one gentleman who spoke very fluent English getting visibly weary during dinner (we'd spent the day together since about 9:00AM), and I asked him if he was OK. He replied that he was getting a headache from having to "think in English" all day.