Sed Regular Expression affecting content after the Regex

Question

I have an HTML file containing the following text:

<!doctype html><html><head><meta charset="utf-8"><title>Test</title><base href="/"><meta name="viewport" content="width=device-width,initial-scale=1"></head><body>test</body></html>

And I run this sed command against it:

sed -i -e "s:<base href\s*=\s*\".*\"\s*>:<base href=\"/apps/test/\">:g" /tmp/test/index.html

I'd expect for that just to replace <base href="/"> with <base href="/apps/test/"> and leave the rest alone, but it ends up affecting content after the regex:

 <!doctype html><html><head><meta charset="utf-8"><title>Test</title><base href="/apps/test/"></head><body>test</body></html>

It ended up removing the entire meta tag found after the regex. Am I just not doing the regex right?

GNU sed version 4.2.1

enjoy: https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags .. see also: https://mywiki.wooledge.org/Quotes and https://stackoverflow.com/questions/5319840/greedy-vs-reluctant-vs-possessive-quantifiers (and sed doesn't support lazy quantifiers) — Sundeep, Mar 08 '18 at 17:16
[Parsing HTML with regexes is a dangerous proposition](https://stackoverflow.com/q/1732348/519360). Doing it "right" requires implementing a full-on HTML parser in regex(es), and while that's technically possible, it's ill advised. …That said, `s:]*>::gi` should work fine as long as the `` tag isn't multi-line. — Adam Katz, Mar 08 '18 at 19:06

Benjamin W. · Accepted Answer · 2021-09-29T08:08:07.533

Because * is greedy, the .* in =\s*\".*\"\s*> matches to the furthest right > available.

You can use single quotes around your command so you don't have to use \" for double quotes. Then, instead of ".*", you can use "[^"]*", which only matches to the next double quote.

This would make your command into

sed 's:<base href\s*=\s*"[^"]*"\s*>:<base href="/apps/test/">:g'

However, manipulating HTML with sed and regexes is eternally brittle and will break at the first possible opportunity. You could use an XML/HTML parser such as xmllint, see Roman's answer; an alternative are the W3C HTML-XML-utils with their hxpipe and hxunpipe commands.

These commands parse your HTML and turn it into a format easily processed with sed, awk & friends, then turn it back into HTML:

$ hxpipe infile.html
!html "" 
(html
(head
Acharset CDATA utf-8
(meta
(title
-Test
)title
Ahref CDATA /
(base
Aname CDATA viewport
Acontent CDATA width=device-width,initial-scale=1
(meta
)head
(body
-test
)body
)html
-\n

so to turn the / in the href for the base tag into /apps/test/, we could do this:

$ hxpipe infile.html \
    | sed '/Ahref CDATA/{N;/\n(base$/s|\(CDATA\) .*|\1 /apps/test/|}' \
    | hxunpipe
<!DOCTYPE html><html><head><meta charset="utf-8"><title>Test</title><meta href="/apps/test/" name="viewport" content="width=device-width,initial-scale=1"></head><body>test</body></html>

where the sed command

sed '/Ahref CDATA/{N;/\n(base$/s|\(CDATA\) .*|\1 /apps/test/|}'

or, better readable

/Ahref CDATA/ {                                # If line matches this
    N                                          # Append next line
    /\n(base$/ s|\(CDATA\) .*|\1 /apps/test/|  # If in base tag, replace href
}

in a more or less robust fashion makes your change.

score 2 · Answer 2 · answered Mar 08 '18 at 17:15

The only right way for processing xml/html data is to use xml/html parsers.

xmlstarlet solution:

xmlstarlet fo -R -H /tmp/test/index.html | xmlstarlet ed -O -u '//base/@href' -v '/apps/test/'

The output:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8"/>
    <title>Test</title>
    <base href="/apps/test/"/>
    <meta name="viewport" content="width=device-width,initial-scale=1"/>
  </head>
  <body>test</body>
</html>

To modify the file in-place add -L option: xmlstarlet ed -L -u ....

Sed Regular Expression affecting content after the Regex

2 Answers2