Derkling's HomePage

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Installing public-inbox ======

Install ''lei'' via the package manager of your distro.

For example, on Arch/Manjaro linux:
<code sh>
sudo pamac install public-inbox
</code>

===== Initialization =====

Initialise the ''lei'' storage by pointing it to a folder where all ''public-inbox''
related files will be stored.

<code sh>
$ export PI_DIR="${HOME}/Mail/.public-inbox"

$ mkdir "${PI_DIR}/lei"
$ lei init "${PI_DIR}/lei"

# /home/derkling/.config/lei/config created
# leistore.dir=/home/derkling/Mail/.public-inbox/lei already initialized''
</code>

The configured ''lei'' storage is tracked by the configuration tile generated by
the previous command:

<code sh>
$ cat ${HOME}/.config/lei/config

[leistore]
	dir = /home/derkling/Mail/.public-inbox/lei
</code>

===== Adding LKML as external repo =====

CLI instructions to clone a public-inbox repo are provided by the LORE page of
each repo. For example, the notes for the LKML repo are available here:
https://lore.kernel.org/lkml/_/text/mirror/

However, we can make use of the ''lei'' CLI to get all done automagically.
The ''LKML'' public inbox can be added as external repo with:

<code sh>
# Setup the extrnal repo to track the last epoch (i.e. ~0)
$ export PI_DIR="${HOME}/Mail/.public-inbox"

$ lei add-external --boost=0 \
  --mirror=https://lore.kernel.org/lkml/ \
  --epoch=~0 \
  "${PI_DIR}/pi-lkml" \
  -j $(nproc)
</code>

which will clone the epoch git repo locally and generate a local index of all
the messages in the cloned epochs.

**NOTE:** The above command clone only the most recent epoch. Tune the ''epoch'' flag
to defined a range if you like to clone older messages.
See https://public-inbox.org/lei-add-external.txt for more options.

This will generate/update the public inbox configuration file, e.g.
<code sh>
$ cat ~/.public-inbox/config

[publicinbox "lkml"]
	address = linux-kernel@vger.kernel.org
	url = http://lore.kernel.org/lkml
	inboxdir = /home/derkling/Mail/.public-inbox/pi-lkml
	newsgroup = org.kernel.vger.linux-kernel''
</code>
====== Quick start with lei CLI ======

For each of the supported ''lei'' command there is a ''lei-.{txt,html}''
documentation file at: https://public-inbox.org/

===== Check configured external sources =====

<code sh>
lei ls-external
</code>

''/home/derkling/Mail/.public-inbox/pi-lkml boost=0''

===== Query for messages =====

The most useful command is ''lei q'' which allows to run all sort of //search// and
it's what powers the [[https://lore.kernel.org/lkml/|LORE's LKML]] web search UI.

For example, to get the full thread related to the last message you sent on
LKML:
<code sh>
lei q "f:$USER" --format text --thread --limit 1
</code>

A full list of the //search terms// available, see the documentation at:
https://public-inbox.org/lei-q.html

===== Export a thread to a Maildir =====

The query command allows to export all the messages matching a query to a
Maildir, which is a quick and convenient way to track upstreams discussion
threads.

For example, to export the full threads related to the last message
posted to LKML with a given keyword in its subject:

<code sh>
KEYWORD='latency_nice'
MDIR=$(mktemp -d /tmp/lei_${KEYWORD:?}_$(date +%Y%m%d_%H%M%S)_XXX)

lei q "s:${KEYWORD:?}" --output "maildir:${MDIR:?}" --thread --limit 10
COUNT=$(find "${MDIR}" | grep cur | wc -l)
echo "Found ${COUNT} messages, available in: ${MDIR}"
</code>

''Found 82 messages, available in: /tmp/lei_latency_nice_20221015_130118_y3h''

===== Appending selected message to a Maildir =====

The query command allows to append results to an existing Maildir. This support
is quite convenient to setup a custom set of filters which allows to distill
from LKML only the messages you really care about.

For example, to keep updating a nominated maildir with all the LKML threads
which, in the last 60 days, have been discussing patches chaging files under
''kernel/sched/'', we can keep periodically running a query such this:

<code sh>
MDIR=~/Mail/lkml/sched
lei q "d:60.days.ago.." "dfn:kernel/sched/*" \
     --threads --output "maildir:${MDIR:?}" \
     --augment
COUNT=$(find "${MDIR}" | grep cur | wc -l)
echo "Found ${COUNT} messages, available in: ${MDIR}"
</code>

''Found 1302 messages, available in: /home/derkling/Mail/lkml/sched''

====== Mutt integration ======

===== Creating personalized per-topic mailboxes =====

By building on top of the example commands provided so far, it's possible to
build a list of ''lei q'' commands which allows for example to filter LKML in
mutliple different ways to:
  * select threads where key contributors are partecipating
  * select messages discussion different subsystems
and have them:
  * appended to dedicated per-topic maildirs
  * appended to a unique filtered maildir
which in both cases will contains only messages we care about.

All the above with the advatage to having a quick and effective way to pull new
messages from the master LORE repo and also to quickly parse and extract the
messages we care about.

The configured maildir(s) can easily be consulted with a simple MUA such as, for
example, ''mutt''.

===== Minimal Mutt configuration =====

TBD

====== Notmuch Integration ======

Start by configuring your email addresses with ''notmuch config''. This generates the
''~/.notmuch-config'' which can be as simple as:
<code>
[database]
path=/home/derkling/Mail

[user]
primary_email=patrick.bellasi@matbug.net
other_email=derkling@gmail.com;patrick.bellasi@gmail.com;

[new]
tags=new;unread
ignore=/.*[.](json|lock|bak)$/;.notmuch;.public-inbox;.mbsyncstate;.isyncuidmap.db

[search]
exclude_tags=deleted;spam

[maildir]
</code>

Hooks can be used to run commands before and after a ''new'' command.

For more details: https://notmuchmail.org/doc/latest/man1/notmuch-config.html

===== Pre-new hook =====

This can be used to pull new message.

<code>
# ~/Mail/.notmuch/hooks/pre-new
#!/bin/bash
mbsync -a
</code>

===== Post-new hook =====

Once messages have been pooled (pre-hook) and notmuch updated the index, a set
of tagging commands can be issued to properly classify messages according to our
needs.

The following approach assumes all new messages pulled have got the ''new'' tag.
This tag is used to triage new messages by assigning them more specialized tags.
At the very end, the ''new'' tag is removed to all messages, thus leaving a clean
state for triage of new messages will be polled later on.

<code>
# ~/Mail/.notmuch/hooks/post-new
#!/bin/bash

# Tag messages based on mail back-end
notmuch tag +gmail -- tag:new folder:gmail/

# Highlight messages sent to me
notmuch tag +to-me -- tag:new to:derkling
notmuch tag +to-me -- tag:new to:patrick.bellasi

# Highlight messages sent to me
notmuch tag +cc-me -- tag:new cc:derkling
notmuch tag +cc-me -- tag:new cc:patrick.bellasi

# Immediately archive all messages from "me"
notmuch tag +sent -- tag:new from:derkling
notmuch tag +sent -- tag:new from:patrick.bellasi

# Mailing lists
notmuch tag +lkml -- tag:new to:linux-kernel
notmuch tag +linux-pm -- tag:new to:linux-pm
notmuch tag +linux-next -- tag:new to:linux-next
notmuch tag +linux-arm-kernel -- tag:new to:linux-arm-kernel

# Finally, retag all "new" messages "inbox" and "unread"
notmuch tag +inbox -new -- tag:new
</code>

===== Usage =====

See https://www.cs.unb.ca/~bremner/scratch/notmuch-doc-wip/notmuch-emacs.html#notmuch-hello
for a complete descrition of the notmuch working modes.

====== B4 Integration ======

https://git.kernel.org/pub/scm/utils/b4/b4.git/about/

<code sh>
python3 -m pip install --user b4
python3 -m pip install --user --upgrade b4
</code>

Expose the binary with either:
<code sh>
alias b4=$HOME/.local/bin/b4
export $PATH=$HOME/.local/bin:$PATH
</code>

===== Example: Pull a series from LORE =====

With just ''b4'' installed you have everything required to pull series directly
from LORE and get them ready to apply them to a local git tree.

For example, assume we like to test the series from this thread:
https://lore.kernel.org/lkml/20220925143908.10846-1-vincent.guittot@linaro.org/
then we can just use these commands:

<code sh>
MESSAGE_ID="20220925143908.10846-1-vincent.guittot@linaro.org"
SERIES_NAME="series_to_test"

b4 am --outdir /tmp --mbox-name "${SERIES_NAME}" "id:${MESSAGE_ID}" 2>&1
</code>

<code>
Analyzing 25 messages in the thread
Checking attestation on all messages, may take a moment...
---
  [PATCH v5 1/7] sched: Introduce latency-nice as a per-task attribute
  [PATCH v5 2/7] sched/core: Propagate parent task's latency requirements to the child task
  [PATCH v5 3/7] sched: Allow sched_{get,set}attr to change latency_nice of the task
  [PATCH v5 4/7] sched/fair: Take into account latency priority at wakeup
  [PATCH v5 5/7] sched/fair: Add sched group latency support
  [PATCH v5 6/7] sched/core: Support latency priority with sched core
  [PATCH v5 7/7] sched/fair: Add latency list
  ---
  Signed: DKIM/linaro.org
---
Total patches: 7
---
Cover: /tmp/series_to_test.cover
 Link: https://lore.kernel.org/r/20220925143908.10846-1-vincent.guittot@linaro.org
 Base: not specified
       git am /tmp/series_to_test.mbx
</code>

The above command generated a ''.mbox'' file named as we requested, which is now
ready to be applied to a kernel tree using the reported ''git'' command:
<code sh>
cd ~/Code/linux && (
    git fetch origin && \
    git checkout v6.0 && \
    git am /tmp/series_to_test.mbx && \
    git lg -n8) 2>&1
</code>

<code>
HEAD is now at 4fe89d07dcc2 Linux 6.0
Applying: sched: Introduce latency-nice as a per-task attribute
Applying: sched/core: Propagate parent task's latency requirements to the child task
Applying: sched: Allow sched_{get,set}attr to change latency_nice of the task
Applying: sched/fair: Take into account latency priority at wakeup
Applying: sched/fair: Add sched group latency support
Applying: sched/core: Support latency priority with sched core
Applying: sched/fair: Add latency list
9d8885fd8762 - sched/fair: Add latency list (HEAD) (2022-10-17 Vincent Guittot)
8b6a2a119f29 - sched/core: Support latency priority with sched core (2022-10-17 Vincent Guittot)
a8b38a1474b2 - sched/fair: Add sched group latency support (2022-10-17 Vincent Guittot)
c5439c18594e - sched/fair: Take into account latency priority at wakeup (2022-10-17 Vincent Guittot)
082707c25c05 - sched: Allow sched_{get,set}attr to change latency_nice of the task (2022-10-17 Parth Shah)
2ff5a5e12bc0 - sched/core: Propagate parent task's latency requirements to the child task (2022-10-17 Parth Shah)
3d4b3b5377cb - sched: Introduce latency-nice as a per-task attribute (2022-10-17 Parth Shah)
4fe89d07dcc2 - Linux 6.0 (tag: v6.0) (2022-10-02 Linus Torvalds)
</code>

====== PIEM Integration ======

Install ''epkgs'' module, which will provide access to the Emacsmirror:

<code bash>
git clone https://github.com/emacsmirror/epkgs.git $HOME/.emacs.d/epkgs
</code>

User manual: https://emacsmirror.net/

===== Browsing EPKGs =====

ELPA provides also a convenient UI to browse Emacsmirror packages.
By installing the ''epkgs'' package, the Emacsmirror will be automagically
cloned under ''~/.emacs.d/epkgs''. This will also give access to a package
browser, but it's not a package manager.

User manual: https://emacsmirror.net/manual/epkg/

===== Installing the PIEM epkg =====

Use the ''epkg-describe-package piem'' to get a list of packages required by the PIEM module.
All the non builtin dependencies will need to be installed in order to use piem.

Both ''piem'' and all its dependencies can be installed by just checking out the
corresponding module from the emacsmirror repo:

<code bash>
# Ensure we use the
git config --global url.https://github.com/.insteadOf git@github.com:


cd $HOME/.emacs.d/epkgs
git submodule init mirror/elfeed mirror/notmuch mirror/transient mirror/piem
git submodule update mirror/elfeed mirror/notmuch mirror/transient mirror/piem
</code>

===== Loading PIEM from your Spacemacs config =====

Ensure first to have the piem package listed as additional packages (not
provided by a layer) spacemacs have to load. This can be done by pointing to
the epkgs mirror folder we just added and updared above.

<code list>
dotspacemacs-additional-packages '(
      (piem :location "~/.emacs.d/epkgs/mirror/piem/")
      epkg)
</code>

Then, we can configure PIEM by setting the public inbox it will work with:
<code lisp>
;; EPKGs
;; User manual: https://emacsmirror.net/manual/epkg/
;; Install the local DB with:
;;    git clone https://github.com/emacsmirror/epkgs.git ~/.emacs.d/epkgs
;; This DB can be updated using the epkg-update command
(require 'piem)
(setq piem-inboxes
      '(("linux-kernel"
	:url "http://lore.kernel.org/lkml"
	:listid "linux-kernel.vger.kernel.org"
	:address "linux-kernel@vger.kernel.org"
	:coderepo "~/Code/linux/")
       ))
</code>

Enable the ingegration with the MUA of choice, e.g. for notmuch:
<code lisp>
(require 'piem-notmuch)
(piem-notmuch-mode 1)
</code>

Finally load all the other CLI tools we will use:
<code lisp>
(require 'piem-b4)
(require 'piem-lei)
(require 'piem-maildir)
</code>

====== Advanced Usage ======

https://public-inbox.org/lei-p2q.html
====== MBSync ======

GMail messages can be synched locally using mbsync. Do that by ensuring to sync
only messages from the "[Gmail]/All Mail" folder since all other folders
generated by GMail as virtual and will have copies of the messages in the all
folder.

Synching only messages from the all folder allows to:
  - sync the read status of messages
  - use notmuch locally to filter / serach messages

This will effectively replicate what GMail does with labels.

You first need to install a keyring CLI to store the IMAP password for your server
<code>
python3 -m pip install --user keyring
</code>


You can get that with the following content in your ''~/.mbsyncrc'':

<code>
### GMail Account
IMAPStore gmail-remote
  Host imap.gmail.com
  User derkling
  # Fetch password from gnome-keyring using keyring Python tool:
  #  pip install keyring
  # Store the password in the keyring with:
  #  keyring set imap_gmail derkling
  PassCmd "keyring get imap_gmail derkling"
  SSLType IMAPS
  SSLVersions TLSv1.2
  Port 993
  AuthMechs Login
  CertificateFile /etc/ssl/certs/ca-certificates.crt
  PipelineDepth 10

MaildirStore gmail-local
  Path  ~/Mail/gmail/
  Inbox ~/Mail/gmail/INBOX
  AltMap yes
  SubFolders Verbatim

# All Mail: all the messages, w/ or w/o the "inbox" label
# Keep only a local cache of 1k archived messages
# ExpireUnread: remove local copy of unread messages
Channel all
  Far    :gmail-remote:"[Gmail]/All Mail"
  Near   :gmail-local:"All"
  Create  Near
  Expunge Both
  Sync    Push Pull All
  SyncState *
  CopyArrivalDate yes
  ExpireUnread yes
  MaxMessages 500

Group gmail all
</code>

The configuration above is good enought to pull new messages via a simple ''mbsync -a'' command.

The sync command can be triggered directly from a notmuch hook, see next section.