Update layout and content by stefan-mueller · Pull Request #115 · quanteda/tutorials.quanteda.io

stefan-mueller · 2026-07-31T07:51:58Z

Content changes

Expand explanations and interpretations on all pages, including literature, interpretation of code outputs, and explanations of the reasoning behind functions
Add a short explainer at the top of each chapter summarising its content
Include relative links between chapters
Add a standalone bib file and cross-references

Technical changes

Standardise all visualisations by using ggplot2
Update website layout

Note: an initial, at times incomplete, draft of the new explanatory text was generated by @claude Code. I then substantively revised and edited this text.

- Include literature as references and create automated reference list in chapters - Improve layout: nicer colours, clearly distinguish between code chunks and code output by using different styles - Adjust header and footer Co-Authored-By: Claude <claude@users.noreply.github.com>

koheiw · 2026-08-05T07:27:51Z

The target branch of this PR should be update-v4.5.0.

koheiw

Looks good overall except the themes. Comments are all minor points.

koheiw · 2026-08-01T23:58:47Z

 print(toks_nopunct)
 ```

+Compare the two tokens objects above: `toks_immig` still contains punctuation marks as separate tokens, while `toks_nopunct` does not, which is usually what you want before counting words. Removing punctuation and numbers at the tokenisation stage, rather than afterwards, is both faster and less error-prone than trying to strip them out later.


Internally, tokens_remove() is used after tokenization.

koheiw · 2026-08-02T00:05:20Z

+
+`what = "word"`, the default you have been using so far, is not a single fixed set of rules, but tracks whichever version of **quanteda**'s built-in word tokenizer is current, set by `quanteda_options("tokens_tokenizer_word")`. That mapping has changed as the package has evolved. For most everyday work this is exactly what you want, since the current tokenizer is also the most accurate one. But it means the same code, `tokens(x)`, can silently produce different tokens after you or a collaborator updates **quanteda**.
+
+`tokens()` also accepts several older, explicitly named versions of the tokenizer, so you can pin down and reproduce a specific rule set rather than relying on whichever version happens to be current. `"word1"` reproduces the pre-version-2 behaviour, `"word2"` and `"word3"` reproduce the versions used in **quanteda** 2 and 3, and `"word4"`, current at the time of writing, is what `what = "word"` currently maps onto.


Be more specific about the version of the package than saying "at the time of writing".

koheiw · 2026-08-02T00:09:17Z

+
+## Inspecting the structure of a tokens object
+
+A tokens object is, underneath its printed display, a list with one element per document, each holding a character vector of that document's tokens in order. Converting it with `as.list()` makes this structure explicit, useful early on for building an accurate mental model of what a tokens object contains.


tokens object is a list of integer vectors.

koheiw · 2026-08-02T00:14:59Z

 draft: false
 ---

+Many terms that matter in social scientific research are not single words but fixed phrases, such as "asylum seeker" or "climate change". If you tokenise text, by default `tokens()` splits these phrases into their separate parts, so "asylum" and "seeker" become two unrelated tokens rather than one meaningful unit.


independent tokens instead of unrelated?

koheiw · 2026-08-02T00:22:45Z

 ```

-To preserve these expressions in a bag-of-word analysis, you have to compound them using `tokens_compound()`.
+Most later analyses, including a document-feature matrix, only count individual tokens and have no way of knowing that two adjacent words belong together. To preserve multi-word expressions like these in that kind of "bag-of-words" analysis, you need to glue them together into a single token first, using `tokens_compound()`. As with `kwic()` in the [previous chapter](/basic-operations/tokens/kwic), `tokens_compound()` only recognises a pattern as a multi-word phrase if you wrap it in `phrase()`. Without it, "asylum seeker" is treated as one long, literal pattern that never matches anything, so the function silently compounds nothing, rather than raising an error.


Do you need "that kind of "?

koheiw · 2026-08-05T07:23:50Z

Please move the changes in layout to separate branch and revert. Applies to all the files under /static and /layouts.

koheiw · 2026-08-06T01:38:03Z

 corp <- corpus_reshape(data_corpus_udhr["cmn_hans"], to = "paragraphs")
-toks <- tokens(corp, remove_punct = TRUE, remove_numbers = TRUE) %>% 
-  tokens_remove(pattern = stopwords("zh_cn", source = "marimo"), min_nchar = 2) %>% 
+toks <- tokens(corp, remove_punct = TRUE, remove_numbers = TRUE) |> 


Good to set concatenator = '' in tokens() in Japanese and Chinese.

koheiw · 2026-08-06T01:39:48Z

-  tokens_remove(pattern = stopwords("ja", source = "marimo"), padding = TRUE) %>% 
+toks <- tokens(corp, remove_punct = TRUE, remove_numbers = TRUE, padding = TRUE) |> 
+  tokens_remove(pattern = stopwords("ja", source = "marimo"), padding = TRUE) |> 
  tokens_select(pattern = "^[ぁ-んァ-ヶー一-龠]+$", valuetype = "regex", padding = TRUE)


Good to set concatenator = '' in tokens() in Japanese and Chinese.

koheiw · 2026-08-06T01:42:49Z

 ```{r}
 # compound collocations
-toks_comp <- tokens_compound(toks, tstat_col[tstat_col$z > 3,], concatenator = "") %>% 
+toks_comp <- tokens_compound(toks, tstat_col[tstat_col$z > 3,], concatenator = "") |> 


If we set concatenator = "" in tokens(), unnecessary.

koheiw · 2026-08-06T01:51:42Z

+theme_set(theme_bw(base_size = 16))
+```
+
+`textmodel_ca()` provides similar functionality to the standalone **ca** package, but the version in **quanteda.textmodels** works directly from a document-feature matrix, so you do not need to build a contingency table by hand first. We return to the Irish budget speeches used for Wordfish in the [previous chapter](/machine-learning/wordfish), so you can compare how the two methods position the same documents.


I don't think ca works with sparse matrices like DFM.

"directly with" instead of "directly from?

koheiw and others added 19 commits July 27, 2026 09:04

Update toml

b51bfdd

Repalce pipe operator

01d5bfa

Ignore temporary files

fb59500

Renive defunction topfeatures.fcm()

2f7541c

Update markdown

2952abe

Update introduction chapters

2b2627a

Update and extend corpus pages

faa5584

Update and extend dfm and fcm

d3aca6f

Significant updates to tokens chapters

1e8015b

Update and extend data import

bdeba90

Update advanced operations chapters

b6cab41

Update statistical analysis chapters

0db3861

Update multilingual pages

e2f61a1

Extend classification, scaling, and topic models

cf79d91

Adjust workflow page

b87b63d

Rebuild website and update figures

e1bfcd8

Adjust colour of note box

8b2fbd9

Hide options(width = 110) in all chapters and rebuild

fb3b091

stefan-mueller requested a review from koheiw July 31, 2026 07:51

koheiw requested changes Aug 6, 2026

View reviewed changes

Navigation Menu

Uh oh!

Update layout and content - #115

Update layout and content#115
stefan-mueller wants to merge 19 commits into
masterfrom
update-layout-and-content

stefan-mueller commented Jul 31, 2026

Uh oh!

koheiw commented Aug 5, 2026

Uh oh!

koheiw left a comment

Uh oh!

koheiw Aug 1, 2026

Uh oh!

koheiw Aug 2, 2026

Uh oh!

koheiw Aug 2, 2026

Uh oh!

koheiw Aug 2, 2026

Uh oh!

koheiw Aug 2, 2026

Uh oh!

koheiw Aug 5, 2026

Uh oh!

koheiw Aug 6, 2026

Uh oh!

koheiw Aug 6, 2026

Uh oh!

koheiw Aug 6, 2026

Uh oh!

koheiw Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		`what = "word"`, the default you have been using so far, is not a single fixed set of rules, but tracks whichever version of quanteda's built-in word tokenizer is current, set by `quanteda_options("tokens_tokenizer_word")`. That mapping has changed as the package has evolved. For most everyday work this is exactly what you want, since the current tokenizer is also the most accurate one. But it means the same code, `tokens(x)`, can silently produce different tokens after you or a collaborator updates quanteda.

		`tokens()` also accepts several older, explicitly named versions of the tokenizer, so you can pin down and reproduce a specific rule set rather than relying on whichever version happens to be current. `"word1"` reproduces the pre-version-2 behaviour, `"word2"` and `"word3"` reproduce the versions used in quanteda 2 and 3, and `"word4"`, current at the time of writing, is what `what = "word"` currently maps onto.


		## Inspecting the structure of a tokens object

		A tokens object is, underneath its printed display, a list with one element per document, each holding a character vector of that document's tokens in order. Converting it with `as.list()` makes this structure explicit, useful early on for building an accurate mental model of what a tokens object contains.

Uh oh!

Conversation

stefan-mueller commented Jul 31, 2026

Content changes

Technical changes

Uh oh!

koheiw commented Aug 5, 2026

Uh oh!

koheiw left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants