You can conditionally load it...
http://scribu.net/wordpress/optimal-script-loading.html
http://pippinsplugins.com/load-scripts-if-post-has-short-code/
http://wordpress.stackexchange.com/questions/108444/how-to-add-stylesheets-only-to-pages-with-specific-shortcode
Thanks for considering doing it this way. Appreciate your plugin.