Silly and paranoid, you think? Maybe so, except...exactly that has been done throughout history. Ad hominem attacks on ideas are part of every social controversy; it's the demagogue's favorite logical fallacy. Can't happen here? It already has. Consider the Alien and Sedition Acts of 1798; Jackson's removal of the Cherokees; Lincoln's suspension of habeas corpus; Prohibition; Roosevelt's imprisonment of Japanese citizens; modern civil forfeiture laws; and of course the Patriot Act. Rights to life, liberty and property are constantly besieged by government. Any government. The Declaration, the Constitution, and the Bill of Rights are our walls against this assault, but they're only as strong as our belief in them and our willingness to fight for them.
Jefferson's ideas, and the centuries of philosophy before them, exist independently of the author's human failings. I wish he'd been a better person, but the ideas count, not the individual. I am equal before the law to any other person. I claim my right to life, liberty and property; to freedom of expression, assembly and worship; to self-defense and due process; and all other rights not specifically enumerated. I do so because I can. I do so because I am willing to fight for these by any means necessary, against anyone who would take them from me. That's it. No more justification is necessary.