Having written about one in a former life, I would say 'test machine'; but I wouldn't say 'testing machine' was wrong - it follows the analogy of many other '-ing' machines ('washing','sorting','mincing'...). It might be useful to decide 'Is it a machine that does it, or a machine that helps a person to do it). A lot of the latter sort are '-er's (even with the word 'test', but only when qualified: 'Have you seen my circuit-tester - it's not where I left it; did you put it back in the box?'). 'Tester' on its own has a special meaning, particularly in the field of cosmetics: a 'tester' is a little bottle of something, used for trying out the contents.; or sometimes it's a full-sized bottle, but it's the only one with the seal broken.
PS A tester is also an old word for a sort of bed, I think. I'll leave the Googling as 'an exercise for the reader'
- For Teachers