Official project website for Datasette, building a search engine with Dogsheep Beta, sqlite-utils analyze-tables

Datasette Weekly(ish) volume 5

Datasette finally has an official project website!

datasette.io is the new home for Datasette. The site features news about the project, explains the problems that Datasette and its ecosystem of tools try to solve and gathers together a complete directory of plugins and tools for using with Datasette.

The site itself is built using Datasette - it’s actually a heavily themed installation of Datasette, making extensive use of the datasette-template-sql plugin and custom pages to implement the different sections of the site. The code for the site is available on GitHub at simonw/datasette.io.

I wrote more about how the site works in datasette.io, an official project website for Datasette.

Adding search to datasette.io

datasette.io also includes a search engine for the project.

It provides faceted search across:

• Every section of the latest documentation (416 total)

• 48 plugin READMEs

• 22 tool READMEs

• 64 news items posted on the Datasette website

• 212 items from my blog

• Release notes from 557 package releases

That’s 1,319 items total! The overall Datasette project continues to grow in all kinds of directions, so having a single search engine to tie it all together felt like a useful addition.

I built the search engine using Dogsheep Beta, the cross-table SQLite search engine I originally built for my Dogsheep Personal Analytics project.

You can read more about how the search engine works, including a breakdown of the YAML configuration used to create the unified search index, in Building a search engine for datasette.io.

sqlite-utils analyze-tables

I released sqlite-utils 3.1 with a useful feature for getting to grips with new data: the “sqlite-utils analyze-tables” command.

Run this against any SQLite database and it will spit out a summary of every column in every table, showing you common and least-common values, the number of distinct items and how many rows contain nulls or blanks.

The analyze-tables documentation has more. In addition to outputting the summary to the command-line you can add the “--save” option to store the generated summary in a SQLite table called _analyze_tables_ - which means you can then further examine it using Datasette.

Here’s an example of the output when running analyze-tables against a database created using my github-to-sqlite tool.

Office hours for 2021

Every Friday I run Datasette Office Hours sessions, where people who use Datasette (or who are interested in using it) can grab a 20 minute video chat with me to talk about the project. I’ve done a couple of weeks of these now and they’ve been fantastic - it’s so interesting hearing how people are using or plan-to-use the tool, and I’ve already had a bunch of great feedback to help me plan the next steps for the project.

If you’re interested in talking to me, slots in January 2021 are available now!