Fixed in revision 8095 by using regexes instead of lxml to parse img tags.

I've just discovered yet another way in which lxml is mangling my HTML. I'm fed up with fixing 
around the edges.

So in the next week or so, I will try re-implement regular expressions to find and replace the 
<img> tags. Given that malthe seems to think it's a reasonable idea I'll do it inside repoze.bitblt 
on a branch first.

So, any ideas on how to robustly find <img> tags in HTML via regexes? Any example 
implementations? 

Also, what kind of backwards compatibility does repoze.bitblt require? Should I replace the 
existing rewrite_image_tags or implement a new one alongside and implement some kind of 
configurability into ImageTransformationMiddleware?

You're right on the slowness of beautifulsoup compared to lxml: 
http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/

Speed matters; ``lxml`` is very fast and does not incur any significant overhead. I feel that 
BeautifulSoup would (this might not be correct).

Regular expressions are very fast, and sometimes brittle. Everything's a compromise. 
However, image-tags are usually the most wellformed elements in an HTML page.

I think it'll work.

As an alternative to regexes, there's always BeautifulSoup http://www.crummy.com/software/BeautifulSoup/documentation.html

I also was bitten by this. Attached is the patch I am using, it includes and expands on the 
originally posted patches using dbaty's "more complex than it should be" method to keep the 
doctype out of html that didn't already have it.

Using regexes does start to seem like a nice idea after looking at all the gymnastics one has to 
go through with lxml.

Perhaps we can use ``lxml`` to extract the locations (string start-
and end- ranges) for the ``<img>`` tags and then simply use regex
matching on those.

This way, the original document isn't changed, but we don't have the
pitfalls of heuristic.

Malthe, I think you have replied to the wrong ticket. The patch I described has
not been applied (and regular expressions, well, we can use them everywhere, of
course, but... ;) )

(Note: I plan to commit the patch next Friday when I have time.)

I see this patch has already been applied.

Perhaps we should consider using regular expressions to do this. Chances are
that it'll be a) faster, b) less intrusive.

When rewriting image tags, repoze.bitblt removes the doctype of any (X)HTML 
content (cf. attached test). It should not.

I have found a fix for XHTML code (cf. attached patch) by changing how the 
content is parsed. However, the bug persists for HTML content (when 'try_html' 
is not enforced). I tried to use the same technique as for XHTML (using 
lxml.etree.parse() instead of lxml.html.document_fromstring()) but the transformed 
content then always includes a doctype. Perhaps we could then remove it when 
it was not present in the original content, but it starts to be a bit more 
complicated than it should... (I admit that I did not dig too much in lxml...)

In a nutshell, the attached patch will keep the doctype for XHTML content. For 
HTML content, the current (bogus) behaviour is kept (and the doctype is 
removed). Malthe (or anyone who uses this package), if you do not object, I'll 
commit the patch.