Books on Monterey - A Followup

In a previous post, I discussed what was wrong with Books for Monterey.  I've been trying to use it since then and, as of today, have given up on the Mac version.  Why?

1. I lost all of my books on the iCloud.  Every one.  iCloud backup was of no use, and I opened a call to apple about this.  After 2 months they still haven't been able to recover them - see below.
2. If you reach for a book and want information on it, and select the 3 dots on the right to look at the "info" to it,  It supplies the cover, title and author.  No collection, date, publisher etc., or any other information.
3. You can't find which collections a book is contained in without going to each collection and looking for it.
4. Collections don' t show the numbers of items in them (as on the iPad version)

I do still use Books, but only on the iPad, but I organise them differently now.

I use Calibre to organise the books I want, tag them etc and then add them to Books when I'm completed.  I use the calibre library and a couple of perl scripts to make sure everything is consistent between calibre and Books.  I would like to be able to create collections via the tags in calibre, but that's for another day.

Audiobooks are still really broken, and I still use Plex and Prologue to listen to them.

Lost iCloud Books

After I installed a new Mac mini, all of my books (over 1500) on iCloud vanished.  On all my devices, totally.

I opened up a call to Apple about this and, as of 2 months, they have not been able to recover them.  The iCloud Drive does *not* save your books.  They're in some strange place.

The local copies of all books are stored in 

<Home directory>/Library/Mobile Documents/iCloud~com~apple~iBooks/Documents/

However, they are not all there.  If the file starts "period"-name.icloud it's just a reference to the file on iCloud and it needs to be downloaded before you can read it.

e.g:.

Backblaze jimmy$ ls -l .1Password\ for\ iOS.epub.icloud 

-rw-r--r--  1 jimmy  staff  173 16 Jan  2016 .1Password for iOS.epub.icloud

Jimmys-Mac:Backblaze jimmy$ hexdump -C .1Password\ for\ iOS.epub.icloud

00000000  62 70 6c 69 73 74 30 30  d3 01 02 03 04 05 06 5c  |bplist00?......\|

00000010  4e 53 55 52 4c 4e 61 6d  65 4b 65 79 5f 10 10 4e  |NSURLNameKey_..N|

00000020  53 55 52 4c 46 69 6c 65  53 69 7a 65 4b 65 79 5f  |SURLFileSizeKey_|

00000030  10 18 4e 53 55 52 4c 46  69 6c 65 52 65 73 6f 75  |..NSURLFileResou|

00000040  72 63 65 54 79 70 65 4b  65 79 5f 10 16 31 50 61  |rceTypeKey_..1Pa|

00000050  73 73 77 6f 72 64 20 66  6f 72 20 69 4f 53 2e 65  |ssword for iOS.e|

00000060  70 75 62 10 00 5f 10 1e  4e 53 55 52 4c 46 69 6c  |pub.._..NSURLFil|

00000070  65 52 65 73 6f 75 72 63  65 54 79 70 65 44 69 72  |eResourceTypeDir|

00000080  65 63 74 6f 72 79 08 0f  1c 2f 4a 63 65 00 00 00  |ectory.../Jce...|

00000090  00 00 00 01 01 00 00 00  00 00 00 00 07 00 00 00  |................|

000000a0  00 00 00 00 00 00 00 00  00 00 00 00 86           |.............|

000000ad

Backblaze jimmy$ plutil -p - < .1Password\ for\ iOS.epub.icloud

{

  "NSURLFileResourceTypeKey" => "NSURLFileResourceTypeDirectory"

  "NSURLFileSizeKey" => 0

  "NSURLNameKey" => "1Password for iOS.epub"

}

If you open up the enclosing file in finder, it lies to you about the files and says it really exists.  Don't believe it.  Finder is lying.

I managed to recover about 1000 from my Backblaze backup, and old disk, a Mac that I hadn't turned on since the files were lost and another Carbon Copy Cloner backup..  The rest I've having to look for slowly on old disks, email, purchases etc.

Dtrace on Monterey

 As I mentioned in a previous post, dtrace was broken in El Capitan and later, but all is not yet lost.  As a post on medium has described, there are other ways around this including using approved methods from Apple.  In normal cases, non system binaries cannot be examined with dtrace.

Chris demonstrates a way to run dtrace in a chroot-ed environment after turning off code signing to allow and common to be run with dtrace as well as any children forked off. 

The officially sanctioned methods of getting security framework events ( not the same as sys call level output) is used be the following programs and can be used to trace some of what dtrace can do, but without the faff:

1. Filemonitor (from Objective See)

Print JSON file on system events pertaining to files.

1. Processmonitor (from Objective See)

Same as above, but for processes

1. Crescendo ( project on GitHub)

Crescendo is a swift based, GUI real time event viewer for macOS. It utilizes Apple's Endpoint Security Framework.

So there is a. bit more hope now... 

MacOS 12 Monterey

 Thoughts on installing Monterey and migrating from Mojave

Music

OK, so everything was asking for iTunes to be split up into different applications, and this is the result.  It now deals with just music, and it's pretty my the same.  I had a few scripts from Doug Scripts which I'll need to change/update to work with this but it looks OK.

Books

Great Maker, what have they done.  This is just awful.  Been running for 2 days and these are my gripes:

1. It's basically an iPad application, let's be honest
2. The list view is not a list.  It's an icon with a title.
3. You can't get information on the files. 'Get Info' retried the title, the author and possibly the cover.  You can't do anything with then, and it's just what you see normally, so what's the point. 
4. It's now the ONLY way to listen to audiobooks.  Previously iTunes stored audio books and could play them.  Now I need to migrate all of the audio books into The Books app.
1. It combines all the files into one title now.  I save a lot of radio plays/episode and this is just horrible.  You can't see home may chapters are in there or re-tag them (sine they are mpeg4 audio files, there are a lot of tags),
2. You can't view all of the tags like, author, description, comment, lyrics, date, Disc/track (which are useful for radio series)
3. The files are stored in ~/Library/Containers/com.apple.BKAgentService/Data/Documents/iBooks/Books/Audiobooks and tagged by a sha1-checksum and not the title, so it's impossible to work out what you have via the filename (the same is true for normal books, but it's still a change from the old method)

I'll be migrating to Plex audio and use Prologue on my iPhone to listen from now on.  I've been using it for a while, on and off, but not this has really convinced me to drop the Apple Ecosystem for my Audio Books/Dramas for certain.  I'll need to change my workflow, but I'll manage.  It used to be 'add to iTunes' and then let Plex look for audiobooks.  Now it'll need to be a bit more involved.  A challenge.

Workaround

To get some semblance or normality for books:

1. Downloaded the Catalina Install image from the App Store.
2. Created a Virtual Machine using VMware Fusion and installed it there.
3. Did almost no setup etc, just to get it running; No Apples ID or anything.
4. Installed VM Ware tools to I could access it better.
5. Located the Binary for Books.app under /Applications and copied it to my Monterey Macintosh and called it 'Cata-Books' instead.
6. Run 'Cata-Books' instead under Monterey to get the old functionality back.
7. Deleted the Catalina VM since it's no longer needed.
8. Breathe again.

AudioBooks

Take everything I said about Books and keep it here, then multiply by a factor.  Whereas previously I could see my Audiobook, edit some metadata, Add new episodes (for radio dramas and comedies), now it's all held under the "improved" Books program.  Can't edit meta-data, can't easily see how the books may be split up into individual files.  Most importantly, I can't specify a separate area to store these like I do for TV and Music.  They *have* to be under ~/Library unless I try something clever with symlinks and external volumes which I really do not want to do.

https://macmost.com/how-to-manage-an-audiobook-collection-in-macos-catalina.html has a good article on this.  Myself, I going to write some code to add files to the correct place based on the metadata tagged, and then  use a combination of Plex and prologue to listen on my iPhone/Ipad.  I started to do this anyway, so I just need to modify the code I wrote to add to iTunes to write to the same place as iTunes used to and they'll be picked up by Plex.  Although Plex isn't that great on searching.

Checkout the following articles as well

• The tragic case of audiobooks
• Audiobooks in the Books app
• Accessing your audio books in MacOS Catalina from reddit
• Why I don’t want to upgrade from Mac OS X 10.14 Mojave

I use MetaZ installed via homebrew to work on tags as well as my own programs to edit the tags.

Come back iTunes.

TV/Movies

Same gripes for iTunes.  List mode is a bit off and you can't separate Downloaded items versus items in the Cloud which is a bit annoying, but not a deal breaker.

PDF support

It looks like you can no longer get around the issue of protected PDFs.  My workflow used to be to get a PDF and Print to Paperless to archive it off to the Paperless program. Under older releases you could:

1. Open the file in Preview
2. Export it as another PDF
3. Close preview.
4. Open the newly created file and then ue the 'Print to Paperless' option or open it withPDFPen and the like.

None of this works any more, so to unprotect a PDF, I need to fire up an non Monterey VM, copy the file in to perform the above and the copy it out again at the end.

Annoyances

1. The Notification Centre is pretty obnoxious.
2. Don't care for 'zsh'.  I installed home-brew and am running bash/ksh from there for all of my needs.  Change my login shell back to bash.
3. Couldn't;t make a new directory called /app or /opt due to the Protection mechanism.  Had to edit /etc/synthetics.conf and add entries for the directories I wanted and then rebooted.  
4. Auidiobook - See above.  Without iTunes there is no built in way to tag both MP4/M4A/M4B and MP3 files with the same program.

Benefits

1. A clean install runs much better than my previous 6 year old base OS, which was upgraded and migrated each time.
2. All of the old Mojave binaries that I did depend on are available in later versions. Homebrew/macports are your friends in this case.
3. Installation of the printer drivers works perfectly well without any of that garbage bundles software that comes with printers.  My Brother Laser printer and Epson Scanner/Printer work perfectly well without having to install any of the vendor supplied software - and is probably more stable as well...

Running a physical Mac Image under Vmware Fusion

A while(!) back, I wrote a document about copying a physical disk image into VMware Fusion.  This was for a older build of the various tools.  Recently I wanted to to the same thing but Using my old Mojave build as a VM and then re-installing from scratch a new version of Monterey and then slowly copying across the items I need and having the VM running Mojave as an emergency backup.

The process from the article Physical Yosemite Machine to VMware fusion Image is just the same, with the following changes.

Firstly, I'm using CCC version 5, which has changed from the version 4 documented on that page.

Changes to the steps on the above page for  Mojave install and CCC 5.0 are as follows:

1. The disk image won't appear as 'not recognised by the computer', but will appear in Disk Utility.  Format the disk as specified, and keep the format as the same.

1. - 22.  When selecting the remote machine you no longer need to copy a package across or change any settings. CCC 5 will check that it can copy files to the remote machine and will alert you as to any changes required.  It will ask for an account and to ensure remote login is turned on.

1. Turn off the networking on the virtual machine setup before rebooting to stop any conflicts with naming etc.  e.g:

• Disable Backblaze backup
• Sign out of iCloud so that Photos, books, calendar, iTunes are not synced unneceesarily
• Disable iTunes Match
• Turn off LittleSnitch and other un-needed services.
• Change the machine name
• Reboot the VM without networking, perform the above and then turn the networking back on in the VMware settings before the 'final' reboot.