How to use sed to extract substring

Question

I have a file containing the following lines:

  <parameter name="PortMappingEnabled" access="readWrite" type="xsd:boolean"></parameter>
  <parameter name="PortMappingLeaseDuration" access="readWrite" activeNotify="canDeny" type="xsd:unsignedInt"></parameter>
  <parameter name="RemoteHost" access="readWrite"></parameter>
  <parameter name="ExternalPort" access="readWrite" type="xsd:unsignedInt"></parameter>
  <parameter name="ExternalPortEndRange" access="readWrite" type="xsd:unsignedInt"></parameter>
  <parameter name="InternalPort" access="readWrite" type="xsd:unsignedInt"></parameter>
  <parameter name="PortMappingProtocol" access="readWrite"></parameter>
  <parameter name="InternalClient" access="readWrite"></parameter>
  <parameter name="PortMappingDescription" access="readWrite"></parameter>

I want to execute command on this file to extract only the parameter names as displayed in the following output:

$sedcommand file.txt
PortMappingEnabled
PortMappingLeaseDuration
RemoteHost
ExternalPort
ExternalPortEndRange
InternalPort
PortMappingProtocol
InternalClient
PortMappingDescription

What could be this command?

Note that you're going to be sad when that XML comes to you on multiple lines, or if the order of the arguments changes. If that's at all a possibility, you'll want to look into using a proper XML parser. — Andy Lester, May 21 '13 at 16:54
Hm, double standard with questions that can be answered in 10 seconds vs. ones that require more time? Where is the post asking what you've tried? Oh wait... — rliu, May 21 '13 at 16:59

score 137 · Answer 1 · answered May 21 '13 at 16:51

137

grep was born to extract things:

grep -Po 'name="\K[^"]*'

test with your data:

kent$  echo '<parameter name="PortMappingEnabled" access="readWrite" type="xsd:boolean"></parameter>
  <parameter name="PortMappingLeaseDuration" access="readWrite" activeNotify="canDeny" type="xsd:unsignedInt"></parameter>
  <parameter name="RemoteHost" access="readWrite"></parameter>
  <parameter name="ExternalPort" access="readWrite" type="xsd:unsignedInt"></parameter>
  <parameter name="ExternalPortEndRange" access="readWrite" type="xsd:unsignedInt"></parameter>
  <parameter name="InternalPort" access="readWrite" type="xsd:unsignedInt"></parameter>
  <parameter name="PortMappingProtocol" access="readWrite"></parameter>
  <parameter name="InternalClient" access="readWrite"></parameter>
  <parameter name="PortMappingDescription" access="readWrite"></parameter>
'|grep -Po 'name="\K[^"]*'
PortMappingEnabled
PortMappingLeaseDuration
RemoteHost
ExternalPort
ExternalPortEndRange
InternalPort
PortMappingProtocol
InternalClient
PortMappingDescription

answered May 21 '13 at 16:51

Kent

189,393
32
233
301

9

Just FYI, from the grep manpage regarding `-P`: "This is highly experimental and **grep -P** may warn of unimplemented features." – Trevor Robinson Dec 03 '14 at 23:20
Not all *nix distros support 'grep -o'. One instance I know of is AIX – Max Mar 31 '15 at 19:36
1

@FukuzawaYukio I think the grep shipped by ubuntu linux should support it right? even though I am not ubuntu user. The question was tagged with Linux & ubuntu, not Unix or Aix. But you comment is correct. – Kent Apr 01 '15 at 07:43
@Kent Right, I forgot to check what the question was targeting. Ubuntu's grep does indeed support `-o`. – Max Apr 01 '15 at 22:54
11

I had to look up `\K`: It keeps what's left of it outside of the match (so you don't get `name="PortMappingLeaseDuration"`. [Further reading](http://www.regular-expressions.info/refadv.html) – nachocab May 19 '16 at 19:53
6

For those not wanting to use the `-P` flag; no other extended regex that is supported by the default grep will do what the `\K` does, but you could simply pipe it through sed: `grep -o 'name="[^"]* | sed 's/name="//g'` – Leon S. Dec 27 '17 at 15:19
4

Alternatively you can also use grep twice: `grep -o 'name="[^"]*' | grep -o '[^"]*$'`. It produces the same result. – Crisu83 May 27 '20 at 12:58
Yea, no `-P` on MacOS: `grep: invalid option -- P` – sh37211 Jan 03 '23 at 21:24
@sh37211 most linux distros preinstall the gnu impl. The question has tagged "Linux" Please check Leon's comment above if -P is not available in your grep impl. – Kent Jan 11 '23 at 22:14
In my case on mac `grep` regex didn't work without `-E` option, so the final command is `grep -Eo 'name="[^"]* | sed 's/name="//g'` – durex Aug 24 '23 at 14:40
@durex the question is tagged with `linux`. On MacOS, if you didn't install related GNU utilities, such as gnu grep, gnu sed, gnu awk. The options could be different from them in Linux. – Kent Aug 25 '23 at 09:39

score 114 · Answer 2 · edited Oct 31 '19 at 23:52

114

sed 's/[^"]*"$[^"]*$.*/\1/'

does the job.

explanation of the part inside ' '

s - tells sed to substitute
/ - start of regex string to search for
[^"]* - any character that is not ", any number of times. (matching parameter name=)
" - just a ".
([^"]*) - anything inside () will be saved for reference to use later. The \ are there so the brackets are not considered as characters to search for. [^"]* means the same as above. (matching RemoteHost for example)
.* - any character, any number of times. (matching " access="readWrite"> /parameter)
/ - end of the search regex, and start of the substitute string.
\1 - reference to that string we found in the brackets above.
/ end of the substitute string.

basically s/search for this/replace with this/ but we're telling him to replace the whole line with just a piece of it we found earlier.

edited Oct 31 '19 at 23:52

Empi3

3
3

answered May 21 '13 at 16:47

unxnut

8,509
3
27
41

68

That's neither simple nor elegant. Just cryptic. – Stefan Aug 22 '14 at 15:00
40

@Stefan, to the untrained eye perhaps. But spend time with RegEx and like jazz or Picasso, you'll appreciate the simple beauty. – SaxDaddy Apr 14 '15 at 19:40
9

That's what cryptic means: completely unreadable to the untrained eye. – John Haugeland Feb 17 '16 at 19:41
50

Thank you for answering the question instead of proposing a different tool! – Jason Suárez May 03 '16 at 00:05
2

ITT: a generation growing up without learning powerful fundamentals like regular expressions! *sigh* Granted, awk, grep, perl, sed, lex, yacc, javascript all use slightly different variations (thanks posix!) but still an extremely powerful programming concept. I'm aghast at the complexity of some of the answers. – PeterT Oct 12 '16 at 20:36
8

While it does the job, it would be beneficial if you actually explain what is going on. – Christian Herenz Dec 19 '17 at 14:12
1

I always forget we need escape ( ) – Sérgio Mar 15 '18 at 01:17
2

Cryptic but insanely powerful. Worth the time learning regexes, believe me. – Thomas Wana Mar 04 '19 at 11:00
The comment saying this is neither simple nor elegant has not read the question. – Carlos Verdes Nov 03 '19 at 18:22
@unxnut This is such a good explain it help me a lot thanks a lot – Huibin Zhang May 06 '20 at 18:15
Thanks mate! You are the best! So many useful info in a single reply =) – Igor Voltaic May 15 '20 at 15:08
perfect tool for the job. at least that how I feel about it given I never understood awk and rarely use grep in an extended way. Also, it's very flexible to be further customized – Boson Bear Jun 27 '20 at 07:57
not sure but from my tests, if it doesn't find a suitable fit, it returns the whole input, no? – Juh_ Aug 10 '23 at 16:09

score 45 · Accepted Answer · answered May 21 '13 at 16:46

45

You want awk.

This would be a quick and dirty hack:

awk -F "\"" '{print $2}' /tmp/file.txt

PortMappingEnabled
PortMappingLeaseDuration
RemoteHost
ExternalPort
ExternalPortEndRange
InternalPort
PortMappingProtocol
InternalClient
PortMappingDescription

answered May 21 '13 at 16:46

Chris

1,410
12
21

3

`cut` will do the job faster :-) – Michał Šrajer May 21 '13 at 17:07
`awk` is link the nuke, when you just need to blow. – Sandburg Apr 08 '22 at 17:05

score 19 · Answer 4 · answered May 21 '13 at 17:06

You should not parse XML using tools like sed, or awk. It's error-prone.

If input changes, and before name parameter you will get new-line character instead of space it will fail some day producing unexpected results.

If you are really sure, that your input will be always formated this way, you can use cut. It's faster than sed and awk:

cut -d'"' -f2 < input.txt

It will be better to first parse it, and extract only parameter name attribute:

xpath -q -e //@name input.txt | cut -d'"' -f2

To learn more about xpath, see this tutorial: http://www.w3schools.com/xpath/

score 13 · Answer 5 · answered Jun 13 '18 at 08:41

13

Explaining how you can use cut:

cat yourxmlfile | cut -d'"' -f2

It will 'cut' all the lines in the file based on " delimiter, and will take the 2nd field , which is what you wanted.

answered Jun 13 '18 at 08:41

Rushi Agrawal

3,208
2
21
26

1

You want to avoid the [useless `cat`](https://stackoverflow.com/questions/11710552/useless-use-of-cat) though. – tripleee Mar 04 '21 at 11:52

How to use sed to extract substring

5 Answers5

Linked

Related