5

Is is possible to install and update Perl (CPAN) modules with universal (x86_64, arm64) architecture support? If yes, then how?

background

On an arm-based macOS computer, a Perl CPAN module can be installed for exactly one designated architecture as follows:

sudo cpan -i Encode
### equivalent since `-arm64` is the native processor in this situation:
sudo arch -arm64 cpan -i Encode

file /Library/Perl/5.30/darwin-thread-multi-2level/auto/Encode/Encode.bundle
# /Library/Perl/5.30/darwin-thread-multi-2level/auto/Encode/Encode.bundle:
#     Mach-O 64-bit bundle arm64

sudo arch -x86_64 cpan -i Encode

file /Library/Perl/5.30/darwin-thread-multi-2level/auto/Encode/Encode.bundle
# /Library/Perl/5.30/darwin-thread-multi-2level/auto/Encode/Encode.bundle:
#     Mach-O 64-bit bundle x86_64

Notice, however, Apple's perl itself is a "universal binary":

file /usr/bin/perl

# /usr/bin/perl: Mach-O universal binary with 2 architectures: 
#    [x86_64:Mach-O 64-bit executable x86_64] 
#    [arm64e:Mach-O 64-bit executable arm64e]
# /usr/bin/perl (for architecture x86_64):  
#     Mach-O 64-bit executable x86_64
# /usr/bin/perl (for architecture arm64e):  
#     Mach-O 64-bit executable arm64e

The XOR of either one architecture or the other presents a conflict when native and non-native applications share the same Perl dependency. For example, GnuCash Finance::Quote does not run on natively on Arm while MacTeX LaTeX Live Update can run natively on either Intel or Arm processors. Both applications use the Pearl Encode module.

The application log error message will be one of the following if the required architecture version is not found:

'/Library/Perl/5.30/darwin-thread-multi-2level/auto/Encode/Encode.bundle' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))

'/Library/Perl/5.30/darwin-thread-multi-2level/auto/Encode/Encode.bundle' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

Note: A workaround for running the applications is to install the common Perl module dependancy for the x86_64 architecture, and then run the universal capable application in Rosetta2 (x86_64) mode.

Additional Findings

cc option '-bundle'

cc -bundle was found in the saved install log.

rm -f blib/arch/auto/Encode/Encode.bundle
cc  -bundle -undefined dynamic_lookup  Encode.o def_t.o encengine.o  -o blib/arch/auto/Encode/Encode.bundle 
chmod 755 blib/arch/auto/Encode/Encode.bundle
…
Manifying 18 pod documents
Files found in blib/arch: installing files in blib/lib into architecture dependent library tree
Installing /Library/Perl/5.30/darwin-thread-multi-2level/auto/Encode/Encode.bundle

However, man cc and cc --help do not provide any developer information about a -bundle option for the cc clang LLVM compiler. So, it's not clear what -bundle is actually doing, or how a Perl novice might use this piece of information.

lipo

It would appear that the "safest way to build Universal binaries is to compile the modules separately and then use lipo to merge the resulting .bundle files." See: meta::cpan Config_u.pm

The Apple article "Building a Universal macOS Binary" provides a such multi-step example:

The following example shows a makefile that compiles a single-source file twice—once for each architecture. It then creates a universal binary by merging the resulting executable files together with the lipo tool.

x86_app: main.c
    $(CC) main.c -o x86_app -target x86_64-apple-macos10.12
arm_app: main.c
    $(CC) main.c -o arm_app -target arm64-apple-macos11
universal_app: x86_app arm_app
    lipo -create -output universal_app x86_app arm_app

lipo requires the individual architecture files as inputs to -create the universal file.

file Encode.bundle

A search and review for all Encode.bundle files found a mixture of universal and non-universal binaries.

find / -name "Encode.bundle"

file /Applications/FreeCAD_0.20.app/Contents/Resources/lib/perl5/5.32/core_perl/auto/Encode/Encode.bundle
file /Library/Perl/5.30/darwin-thread-multi-2level/auto/Encode/Encode.bundle
file /System/Library/Perl/5.30/darwin-thread-multi-2level/auto/Encode/Encode.bundle
file /System/Library/Perl/5.34/darwin-thread-multi-2level/auto/Encode/Encode.bundle
file /Users/USERNAME/.cpan/build/Encode-3.19-0/blib/arch/auto/Encode/Encode.bundle

# /Applications/FreeCAD_0.20.app/…/core_perl/auto/Encode/Encode.bundle: 
#     Mach-O 64-bit bundle x86_64

# /Library/Perl/5.30/darwin-thread-multi-2level/auto/Encode/Encode.bundle: 
#     Mach-O 64-bit bundle x86_64

# /System/Library/Perl/5.30/darwin-thread-multi-2level/auto/Encode/Encode.bundle: 
#     Mach-O universal binary with 2 architectures: 
#         [x86_64:Mach-O 64-bit bundle x86_64] 
#         [arm64e:Mach-O 64-bit bundle arm64e]

# /System/Library/Perl/5.34/darwin-thread-multi-2level/auto/Encode/Encode.bundle:
#     Mach-O universal binary with 2 architectures: 
#         [x86_64:Mach-O 64-bit bundle x86_64]
#         [arm64e:Mach-O 64-bit bundle arm64e]

# /Users/USERNAME/.cpan/build/Encode-3.19-0/blib/arch/auto/Encode/Encode.bundle: 
#     Mach-O 64-bit bundle x86_64

observations:

  1. file /System/Library/Perl/…Encode.bundle shows that universal binary use does indeed exist for Pearl.
  2. file /Library/Perl/…Encode.bundle indicate that a user install and/or update may be masking the universal /System/Library/Perl/…Encode.bundle from shared application use.

objectives

Ideally, an overall solution would:

  1. be generally enough to not require modification of each individual module which gets added.
  2. work for both an initial module installation and any subsequent updates.
  3. does not create dependency conflicts with in the Perl installation.

possible approaches

Whether implicitly or expressly invoked, lipo appear to be needed to create the univeral binary.

Just thinking out-loud about some approach directions:

  • modify the Perl make file? (how would one safely do this? is this a practical approach?)

  • create an updated version of Config_u?

perl -MConfig_u Makefile.PL

  • have parallel /Perl/arm64/… and /Perl/x86_64/… trees which are then lipo merged into some /Perl/some_universal_version/… via a script.

  • Could this be a easy as sudo arch -x86_64 -arm64 -arm64e cpan -i Encode?

marc-medley
  • 8,931
  • 5
  • 60
  • 66
  • 1
    Note that the command used by [ExtUtils::MakeMaker](https://metacpan.org/pod/ExtUtils::MakeMaker) to compile the `Encode` bundle is `cc -mmacosx-version-min=12.0 -bundle -undefined dynamic_lookup -fstack-protector-strong Encode.o def_t.o encengine.o -o blib/arch/auto/Encode/Encode.bundle`. If it is possible to create universal `.bundle` file (see e.g. https://ss64.com/osx/lipo.html) then it should be possible to modify [`Makefile.PL`](https://metacpan.org/release/DANKOGAI/Encode-3.19/source/Makefile.PL) or maybe a change to `ExtUtils::MakeMaker` is also needed to make this work? – Håkon Hægland Apr 12 '23 at 19:59
  • @HåkonHægland `-bundle` is interesting. However, additional developer information on the `-bundle` option was not available on the system itself. `lipo` appears to be the needed tool whether invoke implicitly or directly. additional findings were added to question based on looking at the details you mentions. – marc-medley Apr 13 '23 at 06:17
  • Asked if `ExtUtils::MakeMaker` could be updated to [Generate a Makefile with "Universal Binary" (e.g. arm64, x86_64) directives #445](https://github.com/Perl-Toolchain-Gang/ExtUtils-MakeMaker/issues/445) – marc-medley Apr 23 '23 at 23:47

2 Answers2

4
env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' perl Makefile.PL

make

arch -x86_64 make test
arch -arm64  make test
arch -arm64e make test

(sudo) make install

Observation: I've had runtime issues if only one of arm64 or arm64e was installed. So far, including both arm64 and arm64e with x86_64 has been successful.

Check:

file Encode.bundle
# Encode.bundle: Mach-O universal binary with 3 architectures: 
#     [x86_64:Mach-O 64-bit bundle x86_64] 
#     [arm64e:Mach-O 64-bit bundle arm64]
#     [arm64e:Mach-O 64-bit bundle arm64e]

Approach was found in the historic man perlmacosx page:

Related to this support is the new environment variable ARCHFLAGS, which provides a way to build extensions for different machine and 32/64-bit architectures. The default architecture to build extensions before 10.5 was the (single) architecture of the building machine. In 10.6 and beyond, the default architectures were changed to building 32-bit for both PowerPC and Intel, and 64-bit only for Intel. With ARCHFLAGS, this can be changed to whatever architectures the user wants to build. For example:

% env ARCHFLAGS='-arch i386 -arch x86_64' perl Makefile.PL
% make
% make install

will build only 2-way universal.

The env ARCHFLAGS='…' variable also works with cpan. However, to replace a single-architecture binary with a universal binary of the same version number, then the -f force option is needed. The cpan version comparison is not univeral binary aware.

# ok for initial install or version update
(sudo) env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' cpan -i Encode

# -f force is needed to "reinstall|replace" an existing version
(sudo) env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' cpan -f -i Encode

Caveat: The -f force option does not provide potential useful test result information.

Approach: App::cpanminus

Example for a clean install GnuCash Finance::Quote using cpanm:

sudo env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' cpan App::cpanminus
sudo env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' cpanm Test2
sudo env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' cpanm Finance::Quote
sudo env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' cpanm JSON::Parse

Caveat: This works for a clean install. An existing install will likely need an uninstall-reinstall or forced-install process to replace the non-universal binary.

Use find and file commands to determine which Perl *.bundle modules need to be fixed by uninstall-reinstall or forced-install approaches.

find /Library/Perl -name "*.bundle" | xargs file
# /Library/Perl/5.30/…/Readonly/XS/XS.bundle:
#     Mach-O universal binary with 3 architectures: 
#         [x86_64:Mach-O 64-bit bundle x86_64] 
#         [arm64:Mach-O 64-bit bundle arm64] 
#         [arm64e:Mach-O 64-bit bundle arm64e]
# /Library/Perl/5.30/…/Test/LeakTrace/LeakTrace.bundle: 
#     Mach-O universal binary with 3 architectures: 
# …
#
# /Library/Perl/5.30/…/Date/Simple/Simple.bundle: 
#     Mach-O 64-bit bundle x86_64
# /Library/Perl/5.30/…/Sereal/Encoder/Encoder.bundle:
#     Mach-O 64-bit bundle x86_64

Related

marc-medley
  • 8,931
  • 5
  • 60
  • 66
1

Is is possible to install and update Perl (CPAN) modules with universal (x86_64, arm64) architecture support?

I believe you can update the .bundle file by adding a -arch flag for the relevant architectures to the Makefile.PL. I tested this with Encode compiling the Encode.bundle by hand instead of using ExtUtils::MakeMaker like this (macOS M1, ventura 13.1, homebrew perl version 5.34):

#! /bin/bash
arch_opt="-arch x86_64 -arch arm64"
opt1="-I./Encode -fno-common -DPERL_DARWIN -fno-strict-aliasing"
opt2="-mmacosx-version-min=12.0 -fstack-protector-strong"
opt3="-pipe -DPERL_USE_SAFE_PUTENV -Wno-error=implicit-function-declaration -O3"
opt4=-DVERSION=\"3.19\"
opt5=-DXS_VERSION=\"3.19\"
opt6="-I/opt/homebrew/Cellar/perl/5.34.0/lib/perl5/5.34.0/darwin-thread-multi-2level/CORE"
ccopts="$arch_opt $opt1 $opt2 $opt3 $opt4 $opt5 $opt6"
ldopts="$arch_opt $opt2"
cc -c  $ccopts Encode.c
cc -c $ccopts def_t.c
cc -c $ccopts encengine.c
cc -bundle -undefined dynamic_lookup $ldopts \
   Encode.o def_t.o encengine.o -o blib/arch/auto/Encode/Encode.bundle
file blib/arch/auto/Encode/Encode.bundle

The output of the last command is now:

blib/arch/auto/Encode/Encode.bundle: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit bundle x86_64] [arm64:Mach-O 64-bit bundle arm64]
blib/arch/auto/Encode/Encode.bundle (for architecture x86_64):  Mach-O 64-bit bundle x86_64
blib/arch/auto/Encode/Encode.bundle (for architecture arm64):   Mach-O 64-bit bundle arm64

which shows that the Encode.bundle has been updated to a universal binary with architectures x86_64 and arm64

Håkon Hægland
  • 39,012
  • 21
  • 81
  • 174
  • Thank you. I will check out this approach. Also, I just finished adding some addition information to the question based on your prior comment to the question. – marc-medley Apr 13 '23 at 06:13