How can I input from a file to step thru it as a filesystem

Question

I have a file that has a listing of files/folders with the full paths. I'm looking for a way to be able to run thru this as if it was a filesystem.

I attached a small part of this file (the original is 33MB with over 460,000 lines)

./
./termux/
./termux/apt.tar
./termux/usr/
./termux/usr/lib/
./termux/usr/lib/libthread_db.so.1
./termux/usr/lib/libpython3.11.so.1.0
./termux/usr/lib/libgssapi_krb5.so.2.2
./termux/usr/lib/libTECkit.so
./termux/usr/lib/gnome-settings-daemon-3.0/
./termux/usr/lib/gnome-settings-daemon-3.0/gtk-modules/
./termux/usr/lib/gnome-settings-daemon-3.0/gtk-modules/at-spi2-atk.desktop
./termux/usr/lib/qt/
./termux/usr/lib/qt/mkspecs/
./termux/usr/lib/qt/mkspecs/integrity-armv7/
./termux/usr/lib/qt/mkspecs/integrity-armv7/qplatformdefs.h
./termux/usr/lib/qt/mkspecs/integrity-armv7/qmake.conf
./termux/usr/lib/qt/mkspecs/dummy/
./termux/usr/lib/gnome-settings-daemon-3.0/gtk-modules/canberra-gtk-module.desktop
...
./ubuntu-fs/
./ubuntu-fs/home/
...
./Other/
./Other/temp/
./Other/temp/Month.db
./Other/2021/
./Other/2021/January.txt
./Other/2021/February.txt
...

What I would like to do is something as follows

start with the root folder /

ls should give the listing of root

termux/
ubuntu-fs/
Other/
...

cd termux should take me into the termux folder ls now would list everything in the termux folder

apt.tar
usr/
...

And so on ...

I't doesn't need to work specifically with ls & cd. This is just to explain the idea I'm looking for.

Kamil Maciorowski · Accepted Answer · 2023-03-17T10:08:32.063

This answer will allow you to do the following:

According to the listing, create a hierarchy of directories and regular files in a directory of your choice. This is itself a solution because you will be able to browse the hierarchy with cd+ls, mc or whatever.
(Optionally) Using the hierarchy, create a SquashFS image for convenient mounting (even as a regular user) in the future.

1. Creating a hierarchy

Take this script:

#!/bin/sh -
file="$1"
dir="$2"
mkdir -p -- "$dir" || { echo >&2 'Cannot create directory.'; exit 1; }
<"$file" grep '^././$'    | tr '\n' '\0' | (cd "$dir" && xargs -0 mkdir -p --)
<"$file" grep '^./.[^/]$' | tr '\n' '\0' | (cd "$dir" && xargs -0 touch    --)

Save it as txt2dir, make it executable (chmod +x txt2dir) and use like this:

./txt2dir /path/to/listing /path/to/directory

/path/to/directory may or may not exist yet. It will be created if needed. Specifying a non-empty directory is possible, but probably you would want to use an empty or not-yet-existing directory.

Notes:

/path/to/directory may be /dev/shm/something for performance.
The listing is processed twice. Directories (from lines ending with /) are created first, regular files (from lines not ending with /) are created later. If in the listing some entry for a regular file precedes the entry for its directory, this will still work.
I assume that for every line not ending with / there exists a line ending with / that describes all directories in the path. In other words, when touch tries to create a regular file (e.g. ./foo/bar/baz), all directories it needs are already there because mkdir has created them (./foo/bar/ was in the listing).

If this is not necessarily the case (i.e. if ./foo/bar/ may be missing), replace grep '…' in the line with mkdir with sed -n '\|^\./| s|/[^/]*$||p'. This variant will use all entries to create necessary directories (e.g. ./foo/bar/baz alone will be enough to create ./foo/bar/), so when touch starts, all directories will be there for sure. This will be at the cost of performance.
Your listing uses newline characters as delimiters, but I don't let them enter xargs. With tr I convert them to null characters, then I use xargs -0. xargs -0 is the best way to process entries verbatim, without treating backslashes and quotes specially; it's not portable though. Any portable solution has its quirks and/or downsides, I won't elaborate.
The regular expressions for grep (or sed) deliberately omit lines not starting with ./. This is in case some original pathname contained a newline character and generated more than one line in the listing. Example:
```
./foo/bar/dir with
newline character/baz
```
From the above pair only the first line will be processed. Unfortunate filenames will be truncated, unfortunate directories will appear as regular files; but at least there will be no misleading garbage in the top-level directory (newline character/baz, if created, would be such garbage).

Since ultimately we use xargs -0, a better approach (but I'm not going to implement it) would be to detect such cases and deal with them, keeping newlines that belong to pathnames and converting others to null characters. Still pathnames with newline-dot-slash substring would deceive even this.

Hopefully the hierarchy you created your listing from did not contain pathnames with newline characters.

And here you go. A browsable hierarchy is in /path/to/directory. When no longer needed, you can remove it with rm -r /path/to/directory. Keep the script and the listing and restore the hierarchy anytime; or proceed with SquashFS (keep reading).

2. Creating and using a SquashFS image

Having a hierarchy created with the above script, you can now create a SquashFS image. If the hierarchy is in /path/to/directory, the command to create an image is:

mksquashfs /path/to/directory image.sqfs

Now you can mount the filesystem:

as root with

 mount -t squashfs image.sqfs /path/to/mountpoint

or as a regular user with
```
 squashfuse image.sqfs /path/to/mountpoint
```
(this requires FUSE to be supported and enabled in your OS).

And now you can browse /path/to/mountpoint. The hierarchy in /path/to/directory is no longer needed. Keep the image and mount/unmount it at will.

Notes:

To unmount after mount use umount /path/to/mountpoint as root; to unmount after squashfuse use fusermount -u /path/to/mountpoint as the user.
squashfuse uses FUSE. See these:
- Why FUSE might be considered insecure on server?
- How to mount FUSE so that all users will have access to it?
Due to compression, the image will most likely be smaller than the listing. If convenient browsing is all you wanted then you may delete the listing and work with the image only.

Addendum

This is the command I used to create a listing that resembles your listing:

( cd /path/to/hierarchy && find . ! -type d -print -o -exec printf '%s/\n' {} \; ) > listing

I also used it for creating listings from hierarchies created from the original listing. This and diff <(sort listing) <(sort listing2) in Bash allowed me to test if my script works well.

choroba · Answer 2 · 2023-03-17T21:38:38.213

Here is a proof of concept of a simple Perl program that does what you need (but I didn't handle all the possible edge cases). Give it the input file name as an argument.

#!/usr/bin/perl
use warnings;
use strict;
use feature qw{ say };
sub dir :lvalue {
    my ($fs, @dirs) = @_;
    return '.' unless @dirs;
    return $fs->{ $dirs[0] } if 1 == @dirs;
    return dir($fs->{ $dirs[0] }, @dirs[1 .. $#dirs])
}
my %fs;
while (<>) {
    chomp;
    my $is_dir = m{/$};
    my @dirs = split m{/};
    my $last = pop @dirs;
    if ($is_dir) {
        dir(%fs, @dirs, $last) = {};
    } else {
        dir(%fs, @dirs)->{$last} = undef;
    }
}
my $current = '.';
sub ls {
    my @args = @_;
    @args = ("") unless @args;
    for my $arg (@args) {
        if ($arg !~ m{^/}) {
            substr $arg, 0, 0, "$current/";
        }
        my @dirs = updirs(split m{/}, $arg);
        for my $member (sort keys %{ dir(%fs, @dirs) }) {
            print $member;
            say dir(%fs, @dirs)->{$member} ? '/' : "";
        }
    }
}
sub cd {
    my @args = @_;
    warn "One argument expected.\n" unless 1 == @args;
    my @dirs = updirs(split m{/},
                      ($args[0] =~ m{^/}) ? $args[0] : "$current/$args[0]");
    if (ref dir(%fs, @dirs)) {
        $current = join '/', @dirs;
    } else {
        warn "Non-existent path.\n";
    }
}
my %dispatch = (ls   => &amp;ls,
                cd   => &amp;cd,
                exit => sub { exit });
while (print({*STDERR} "$current: "), $_ = <STDIN>) {
    chomp;
    my ($command, @args) = split;
    warn "Unknown command '$command'.\n"
        unless exists $dispatch{$command};
    $dispatch{$command}->(@args);
}
sub updirs {
    my @dirs = @_;
    shift @dirs if "" eq $dirs[0];
    for (my $i = 0; $i <= $#dirs; ++$i) {
        if ($dirs[$i] eq '..') {
            if ($i > 0) {
                splice @dirs, $i - 1, 2;
                $i -= 2;
            } else {
                splice @dirs, 0, 1;
                $i = -1;
            }
        }
    }
    return @dirs
}

How can I input from a file to step thru it as a filesystem

2 Answers2

1. Creating a hierarchy

2. Creating and using a SquashFS image

Addendum