What do you wear to an interview with a temp agency?

I had one interview with one a few years ago and went in a business suit. There were about four other people there interviewing, all in jeans and a casual top. I know it's better to overdress for an interview, but can I easily get away with a simple skirt/khakis and a nice top rather than something as formal as a suit? Thanks!