It was the was the way his eyes were always in shadow under his hat and the fact that he seemed to always be hiding some knowledge from Ichigo and crew. He seemed kind of shady at first.
After we learned he created th Hogyoku I stopped thinking he was up to something shady though and he was more concerned with correcting the trouble The Hogyoku caused thanks to Aizen.