"When i come to visit do what you have to do, i won't keep you. Don't let me being there hold you up."
Thanks
Apart from the fact that you need to capitalise "I", there's really nothing wrong with it. My only suggestion would be "Don't let me (some people will say "my") being there get in your way" instead of "hold you up".